Hi Leonard,
Thanks for your reply.
> 1. a How to construct a CatalogDescriptor ?
I think it would be helpful to add a method for constructing a
CatalogDescriptor, as you mentioned in 1.c. I will update the documentation
later.
> 1.b How to visit the fields ? Could we use Configuration instead of
Map<String, String> ?
I believe that the use of Map<String, String> options is only intended for
creating a catalog and not for accessing internal parameters.
Since all of the relevant parameters for CREATE CATALOG are also stored in
Map<String, String> options, my understanding is that using Map<String,
String> options should suffice.
Here is the implementation of execute CREATE CATALOG statement.
```java
private TableResultInternal createCatalog(CreateCatalogOperation operation)
{
String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
try {
String catalogName = operation.getCatalogName();
Map<String, String> properties = operation.getProperties();
Catalog catalog =
FactoryUtil.createCatalog(
catalogName,
properties,
tableConfig,
resourceManager.getUserClassLoader());
catalogManager.registerCatalog(catalogName, catalog);
return TableResultImpl.TABLE_RESULT_OK;
} catch (CatalogException e) {
throw new ValidationException(exMsg, e);
}
}
```
> 2. Do we have plan to offer a default CatalogStore if user didn’t config
this?
Yes, the in-memory catalogStore will be used as the default CatalogStore
even if the user has not configured one
Best,
Feng
On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xbjt...@gmail.com> wrote:
> Hi, Feng
>
> Sorry for reply late, but I’ve some comments about the FLIP
>
>
> 1. The introduced Public class CatalogDescriptor seems missed some
> necessary component
> a) How to construct a CatalogDescriptor ?
> b) How to visit the fields ? Could we use Configuration instead of
> Map<String, String> ?
> c) Could we offer a built-in factory method to build a CatalogDescriptor
> like
> public static CatalogDescriptor of(String catalogName, Configuration
> configuration)
>
> 2. The FLIP said “By default, there are two built-in CatalogStores
> available: the In-Memory CatalogStore and the File CatalogStore” ,
> Do we have plan to offer a default CatalogStore if user didn’t config
> this? IIUC, users can obtains the benefits from lazy catalog
> initialization If
> we have a default catalogstore even it is in-memory.
>
> Best,
> Leonard
>
>
>
> > On Jun 6, 2023, at 7:08 PM, Feng Jin <jinfeng1...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > Thank you everyone for your valuable input. If there are no further
> > questions or concerns about the FLIP[1], I would like to start voting
> > tomorrow (6/7).
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Best,
> > Feng
> >
> >
> > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <jinfeng1...@gmail.com> wrote:
> >
> >> Hi Samrat,
> >>
> >> Thanks for your advice.
> >>
> >>> 1. The createCatalog method does not mention any exceptions being
> >> thrown.
> >>
> >> CreateCatalog will throw CatalogException like registerCatalog. As
> >> CatalogException is a RuntimeException,
> >> there is no explicit declaration of throwing Exceptions in
> CatalogManager
> >> and TableEnvironment.
> >> To avoid misunderstandings, I have added the "throw CatalogException"
> flag
> >> to the createCatalog method definition of CatalogStore.
> >>
> >>> 2. Could you please provide an exhaustive list of the supported kinds?
> >>
> >> Sure, the documentation now includes the configuration of the built-in
> >> CatalogStore as well as how to configure a custom CatalogStore.
> >>
> >>
> >> Best,
> >> Feng
> >>
> >>
> >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <decordea...@gmail.com>
> wrote:
> >>
> >>> Hi Feng,
> >>>
> >>> Thank you for providing the proposal. I believe this feature will be
> >>> highly
> >>> valuable.
> >>>
> >>> I have a couple of inquiries:
> >>>
> >>> 1. According to the documentation [1], the createCatalog method does
> not
> >>> mention any exceptions being thrown. However, I would like to confirm
> if
> >>> it
> >>> is always true that there will be no failures in all scenarios. Please
> let
> >>> me know if there is any additional information I may have missed.
> >>>
> >>> 2. Regarding the registration process using the
> `table.catalog-store.kind`
> >>> configuration, could you please provide an exhaustive list of the
> >>> supported
> >>> kinds?
> >>> It would be great to have a comprehensive understanding of the
> options
> >>> available.
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>
> >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <jinfeng1...@gmail.com> wrote:
> >>>
> >>>> Hi Hang and Jark
> >>>>
> >>>> Thank you very much for your reply.
> >>>>
> >>>> @Jark
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>> Interface" section?
> >>>>
> >>>> Indeed, it is more reasonable.
> >>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>>>
> >>>> Sure, the names that you provided are indeed more logical and easier
> >>> to
> >>>> understand.
> >>>>
> >>>>> 3. About the open/close method in CatalogStoreFactory.
> >>>>
> >>>> I also agree with the proposed requirement scenario and design. I have
> >>>> already made modifications to the interface.
> >>>>
> >>>> @Hang
> >>>>
> >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> close
> >>>> the resource.
> >>>>
> >>>> I have already added the missing method.
> >>>>
> >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>
> >>>> Sorry for the misspelling, I updated this part.
> >>>>
> >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> methods
> >>>> to init or close its resource
> >>>>
> >>>> This is indeed a scenario that needs to be considered, and I have
> added
> >>>> relevant methods accordingly.
> >>>> Additionally, I have included some relevant description information in
> >>> the
> >>>> documentation to help others understand
> >>>> why the open/close methods need to be added.
> >>>>
> >>>>
> >>>> Best,
> >>>> Feng
> >>>>
> >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <imj...@gmail.com> wrote:
> >>>>
> >>>>> Hi Jing,
> >>>>>
> >>>>> Thank you for the update.
> >>>>>
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>>> Interface" section?
> >>>>> "Proposed Changes" is more like a place to describe the
> implementation
> >>>>> details.
> >>>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>> Besides,
> >>>>> the config key
> >>>>> name should be hierarchical[1]. Therefore, it may be better to use:
> >>>>> "table.catalog-store.kind"
> >>>>> "table.catalog-store.file.path"
> >>>>>
> >>>>> 3. I think Hang's suggestions make sense.
> >>>>>
> >>>>> Others look good to me.
> >>>>>
> >>>>> Best,
> >>>>> Jark
> >>>>>
> >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ruanhang1...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi, Feng.
> >>>>>>
> >>>>>> Thanks for the update.
> >>>>>> The current design looks good to me. I have some minor comments.
> >>>>>>
> >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> >>> close
> >>>>> the
> >>>>>> resource. For example, when storing the information in MySQL, the
> >>> store
> >>>>>> needs to open and close the connections.
> >>>>>>
> >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>>>
> >>>>>> 3. About the usage in the sql gateway.
> >>>>>> Considering the usage in sql gateway, the sql gateway may create a
> >>>>>> CatalogStore for each session.
> >>>>>> If we are using the MySqlCatalogStore, there would be so many
> >>>>> connections.
> >>>>>> How can we reuse the connection among these sessions?
> >>>>>> I think sql gateway need to maintain a connection pool in
> >>>>>> the CatalogStoreFactory and each session get its own connection from
> >>>> the
> >>>>>> pool when it is created.
> >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
> >>> to
> >>>>> init
> >>>>>> or close its resource.
> >>>>>> Or is there a better way?
> >>>>>>
> >>>>>> Best,
> >>>>>> Hang
> >>>>>>
> >>>>>> Feng Jin <jinfeng1...@gmail.com> 于2023年6月2日周五 14:45写道：
> >>>>>>
> >>>>>>> Thanks Jingsong.
> >>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> >>>>>>>
> >>>>>>> +1 For this, I have already updated FLIP.
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Feng
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> >>> jingsongl...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Feng,
> >>>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> >>>>>>>> functions are converted to createxxx from registerxxx.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1...@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi jark, thanks for your suggestion.
> >>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> >>>>> Otherwise,
> >>>>>>>> there
> >>>>>>>>> might be some data inconsistencies when CatalogStore is
> >>> changed.
> >>>>>>>>>
> >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> >>>>>> TableEnv.
> >>>>>>>>> Therefore, my current proposal is to add a method to configure
> >>>> the
> >>>>>>>>> CatalogStore in EnvironmentSettings.
> >>>>>>>>>
> >>>>>>>>> // Initialize a catalog Store
> >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> >>>>>>>>>
> >>>>>>>>> // set up the Table API
> >>>>>>>>> final EnvironmentSettings settings =
> >>>>>>>>>
> >>> EnvironmentSettings.newInstance().inBatchMode()
> >>>>>>>>> .withCatalogStore(catalogStore)
> >>>>>>>>> .build();
> >>>>>>>>>
> >>>>>>>>> final TableEnvironment tableEnv =
> >>>>>>>> TableEnvironment.create(settings);
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>> not an interface method?
> >>>>>>>>>
> >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> >>> that
> >>>> as
> >>>>> a
> >>>>>>> new
> >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> >>>> method
> >>>>>> but
> >>>>>>>> an
> >>>>>>>>> interface method. I have already modified the interface.
> >>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>>>>>> deprecated`registerCatalog`.
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>> other
> >>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>
> >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1].
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Feng
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <imj...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Feng,
> >>>>>>>>>>
> >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> >>>>>>>>>> The current design looks pretty good to me. I just have some
> >>>>> minor
> >>>>>>>>>> comments.
> >>>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>> CatalogStore
> >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> >>> there
> >>>>>> might
> >>>>>>> be
> >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> >>>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>>> not an interface method?
> >>>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>> deprecated
> >>>>>>>>>> `registerCatalog`.
> >>>>>>>>>>
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>>> other
> >>>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Jark
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> >>> jinfeng1...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> >>>>>>>>>>>
> >>>>>>>>>>> @Ron
> >>>>>>>>>>>
> >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> >>>> method,
> >>>>>> do
> >>>>>>> we
> >>>>>>>>>> need
> >>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, we will provide a default
> >>> InMemoryCatalogStoreFactory to
> >>>>>>> create
> >>>>>>>> an
> >>>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>
> >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> >>> initializing
> >>>>> it,
> >>>>>>>> whether
> >>>>>>>>>> we
> >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> >>> snapshots,
> >>>>> and
> >>>>>>>> once
> >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs.
> >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> >>> the
> >>>>>> primary
> >>>>>>>>>> source
> >>>>>>>>>>> for obtaining the corresponding Catalog.
> >>>>>>>>>>>
> >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> >>>>>>>>>>> `catalog.store.path`?
> >>>>>>>>>>>
> >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> >>> seem a
> >>>>> bit
> >>>>>>>>>> strange. I
> >>>>>>>>>>> will update the FLIP.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> @Shammon
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you for the review. I have made the necessary
> >>>>> corrections.
> >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> >>> I
> >>>>> have
> >>>>>>> also
> >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Feng
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> >>>> zjur...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi feng,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for updating, I have some minor comments
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> >>>>>> `Public
> >>>>>>>>>>>> Interfaces`, it is not a public interface.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> >>> and
> >>>>>>>>>>>> `CatalogStoreFactory`
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> >>>>> should
> >>>>>> be
> >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);`
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> >>>> ron9....@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Feng
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> >>>> useful
> >>>>>> for
> >>>>>>>>>> catalog
> >>>>>>>>>>>>> management.
> >>>>>>>>>>>>> I have some small questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1. Regarding the
> >>> CatalogStoreFactory#createCatalogStore
> >>>>>> method,
> >>>>>>>> do we
> >>>>>>>>>>>> need
> >>>>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> >>>> initializing
> >>>>>> it,
> >>>>>>>>>> whether
> >>>>>>>>>>> we
> >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> >>> them to
> >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Feng Jin <jinfeng1...@gmail.com> 于2023年5月29日周一
> >>> 21:19写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi yuxia
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> >>>>> register
> >>>>>>> the
> >>>>>>>>>>> Catalog,
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>> initialize it and open it. right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> >>>>>> original
> >>>>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> >>>>>>>>>> registerCatalog
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> that parameter validation can be performed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> >>>>>>> reflected
> >>>>>>>> in
> >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> >>>>> catalog
> >>>>>>>>>>>>> configurations,
> >>>>>>>>>>>>>> only initialization is required in getCatalog.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> >>>>>>>> luoyu...@alumni.sjtu.edu.cn
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Feng.
> >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> >>>>>>>> initialization*.
> >>>>>>>>>>> If
> >>>>>>>>>>>>> i'm
> >>>>>>>>>>>>>>> wrong, please correct me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> >>>> access
> >>>>>> the
> >>>>>>>>>>> catalog,
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> >>>>> Changes,
> >>>>>>>> once we
> >>>>>>>>>>>>>> register
> >>>>>>>>>>>>>>> the Catalog,
> >>>>>>>>>>>>>>> we initialize it and open it. right?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>> 发件人: "Jing Ge" <j...@ververica.com.INVALID>
> >>>>>>>>>>>>>>> 收件人: "dev" <dev@flink.apache.org>
> >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> >>>>>> Catalog
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>> and asynchronous registration
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> One of the major changes is that current design
> >>> will
> >>>>>>> provide
> >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> >>>> of a
> >>>>>>>> cache,
> >>>>>>>>>>> which
> >>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> >>>> other
> >>>>>>>> sessions
> >>>>>>>>>>> will
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> >>>>>> further
> >>>>>>>>>>>> improvement.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> >>>>>>>> jinfeng1...@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> >>>>>> progress
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> >>> Shammon
> >>>> FY,
> >>>>>>>> ShengKai
> >>>>>>>>>>> Fang
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> >>>>>> solution
> >>>>>>>> for
> >>>>>>>>>>> Flink
> >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> >>>> have
> >>>>>>>> updated
> >>>>>>>>>> the
> >>>>>>>>>>>>> final
> >>>>>>>>>>>>>>>> solution in FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> >>>>>>>> configuration
> >>>>>>>>>>>> similar
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> TableDescriptor.
> >>>>>>>>>>>>>>>> 2.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> >>>>>>>>>>> storeCatalog(String
> >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> >>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>> getCatalog(String)
> >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> >>>> Catalog
> >>>>>>>> instance.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> >>> saving
> >>>>> and
> >>>>>>>>>>> retrieving
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> configurations without having to initialize
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>> 3.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The default registerCatalog(String
> >>> catalogName,
> >>>>>>> Catalog
> >>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> >>>>>>> deprecated.
> >>>>>>>>>>>>>>>> 4.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> A new function registerCatalog(String
> >>>> catalogName,
> >>>>>>>>>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> >>> default
> >>>>>>>> registration
> >>>>>>>>>>>>> function
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> >>>>>>>>>>>>>>>> 5.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> >>>> will
> >>>>>>>> remain
> >>>>>>>>>>>>> unchanged
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> >>>> deletion
> >>>>>>>>>> operations
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>> session won't synchronize with other
> >>> sessions.
> >>>>>>>>>>>>>>>> 6.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> To support multi-session synchronization
> >>>> scenarios
> >>>>>> for
> >>>>>>>>>>> deletions
> >>>>>>>>>>>>>> later
> >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> >>>>>>>>>>> configurable.There
> >>>>>>>>>>>>> may
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> three possible situations:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> >>>>>>>> automatically
> >>>>>>>>>>> clear
> >>>>>>>>>>>>>>>> long-unused catalogs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> >>>>>> getCatalog
> >>>>>>>>>>> creates a
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the document for discussion:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the final proposal document:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you very much for your attention and
> >>>>> suggestions
> >>>>>> on
> >>>>>>>> this
> >>>>>>>>>>>>> FLIP. A
> >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> >>> on
> >>>>> the
> >>>>>>>> entire
> >>>>>>>>>>>> design
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> organizing offline discussions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> >>>> about
> >>>>>>> this
> >>>>>>>> FLIP
> >>>>>>>>>>>>> please
> >>>>>>>>>>>>>>> feel
> >>>>>>>>>>>>>>>> free to share.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> >>>>>>>>>>> <j...@ververica.com.invalid
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> >>> to
> >>>> me.
> >>>>>> We
> >>>>>>>> could
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> >>>> common
> >>>>>>>> built-in
> >>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> >>> every
> >>>>>>>>>> CatalogSotre
> >>>>>>>>>>>>>>>>> implementation has to take care of it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> >>>>>>>>>> jinfeng1...@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jing,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> >>> the
> >>>>> same
> >>>>>>>> time, I
> >>>>>>>>>>>>> greatly
> >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> >>>> thoroughly
> >>>>>>>> checked
> >>>>>>>>>> the
> >>>>>>>>>>>>>> entire
> >>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> >>> FLIP
> >>>>>>> design,
> >>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> >>> being
> >>>>>>>> configured.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> >>> was
> >>>> not
> >>>>>>>> necessary
> >>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> >>>> catalogs
> >>>>>> to
> >>>>>>>> store
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> >>>>>>> Therefore, I
> >>>>>>>>>>>> modified
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> >>> Catalog>
> >>>>>>> catalogs
> >>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> CatalogManager.
> >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> >>> serve
> >>>> as
> >>>>>> the
> >>>>>>>>>> default
> >>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> >>>>>>>> functionality of
> >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> >>>>>>>>>>>>>>>>>> The previous plan that kept
> >>>>>>>> Map<String,Catalog>catalogs has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>> moved
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>> <j...@ververica.com.invalid
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There are still many places contain
> >>>>> inconsistent
> >>>>>>>> content,
> >>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> >>> used.
> >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> >>> not
> >>>>> tell
> >>>>>>>> what the
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> doing.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> >>> issues.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> >>> mean
> >>>>> that
> >>>>>>> you
> >>>>>>>> will
> >>>>>>>>>>>> build
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> >>> is a
> >>>>> very
> >>>>>>>>>> different
> >>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> >>> I am
> >>>>> not
> >>>>>>>>>> mistaken,
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> >>>> work
> >>>>>>>> without
> >>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> >>>>> reason
> >>>>>>>> why I
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> >>>>> Optional
> >>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> correct.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> >>>>>>>>>>>>> jinfeng1...@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Jing
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> >>>> FLIP
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> >>>> review. I
> >>>>>>> have
> >>>>>>>>>>> already
> >>>>>>>>>>>>> made
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> relevant changes.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> >>> the
> >>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>> scenarios, I
> >>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> >>> multiple
> >>>>>>>> scenarios
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> >>>>> remove
> >>>>>>>>>>> 'Map<String,
> >>>>>>>>>>>>>>>> Catalog>'
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> >>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> >>>>> achieve
> >>>>>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> >>>> think?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>>>> <j...@ververica.com.invalid
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> >>> are
> >>>>>> still
> >>>>>>>> some
> >>>>>>>>>>> NIT
> >>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>> like:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> >>>> has
> >>>>>> been
> >>>>>>>> used
> >>>>>>>>>> as
> >>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> >>> example.
> >>>> It
> >>>>>>>> should be
> >>>>>>>>>>>> fine
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> >>> after
> >>>> you
> >>>>>>>> submit
> >>>>>>>>>> the
> >>>>>>>>>>>> PR.
> >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> >>>> somewhere
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>>> rejected
> >>>>>>>>>>>>>>>> section
> >>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> >>> it
> >>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> >>>>>>>> multi-instance
> >>>>>>>>>>>>>>> scenarios,
> >>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> >>> behaviour.
> >>>> If
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> >>>> once
> >>>>>> the
> >>>>>>>>>> related
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> >>> another
> >>>>>>>> instance.
> >>>>>>>>>> Did
> >>>>>>>>>>> I
> >>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>> something?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> >>> Jin
> >>>> <
> >>>>>>>>>>>>>>> jinfeng1...@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Jing
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> >>> name, I
> >>>>>> have
> >>>>>>>>>> updated
> >>>>>>>>>>> it
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> documentation.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> >>> should
> >>>> be
> >>>>>>>>>> implemented
> >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> >>>>>> alternative
> >>>>>>> 1
> >>>>>>>>>> turns
> >>>>>>>>>>>> out
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> >>> this
> >>>>> part
> >>>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> >>>> forward,
> >>>>>> the
> >>>>>>>>>> option 2
> >>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> >>>>> offline,
> >>>>>>> we
> >>>>>>>>>> think
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> >>>>>>>> inconsistency
> >>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> >>>>> descriptions
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> >>>> document
> >>>>>> and
> >>>>>>>>>>>> appreciate
> >>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> >>> in
> >>>>>>>> `Rejected
> >>>>>>>>>>>>>>> Alternatives`
> >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> >>> `Rejected
> >>>>>>>>>> Alternatives`.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best
> >>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> >>> Shammon
> >>>>> FY
> >>>>>> <
> >>>>>>>>>>>>>>> zjur...@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> >>>>>> `Proposed
> >>>>>>>>>>> Changes`,
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>> put
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> >>>>>>>> Alternatives`? I
> >>>>>>>>>>> think
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>> us
> >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> >>> Jing
> >>>> Ge
> >>>>>>>>>>>>>>>>>> <j...@ververica.com.invalid
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> >>> am
> >>>>>> still
> >>>>>>>>>> confused
> >>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>>>>>>>>>> reading
> >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> >>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> >>>>> registerCatalog()
> >>>>>>> and
> >>>>>>>>>>>>>> addCatalog()
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>>> big
> >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> >>>> One
> >>>>> of
> >>>>>>>> them is
> >>>>>>>>>>>>>>> responsible
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> >>>>>> persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> >>>> Map<String,
> >>>>>>>> Catalog>
> >>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>> as a
> >>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> >>> data
> >>>>>>>> persistence.
> >>>>>>>>>> I
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> >>>>> clearly
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>> Some
> >>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> >>>> i.e.
> >>>>>> data
> >>>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> >>>> instance
> >>>>>>>> should be
> >>>>>>>>>>>> found
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> >>>>>> initialized
> >>>>>>>> or it
> >>>>>>>>>>> will
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>> initialized)
> >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> >>>>>>> consistency
> >>>>>>>> will
> >>>>>>>>>>> be
> >>>>>>>>>>>>>> taken
> >>>>>>>>>>>>>>>> care
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> while
> >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> >>>>>> forward,
> >>>>>>>> the
> >>>>>>>>>>>> option
> >>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> >>>>>> because,
> >>>>>>>> afaiu,
> >>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> >>>>> solved
> >>>>>>>> yet. Do
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> propose
> >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> >>>> suggestions
> >>>>>> for
> >>>>>>>> both
> >>>>>>>>>> of
> >>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> >>>> there
> >>>>>> are
> >>>>>>>> some
> >>>>>>>>>>>>>>> inconsistent
> >>>>>>>>>>>>>>>>>>>>> descriptions
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> >>>> clean
> >>>>>>> them
> >>>>>>>> up?
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> >>> Feng
> >>>>>> Jin <
> >>>>>>>>>>>>>>>>> jinfeng1...@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> >>> and
> >>>> I
> >>>>>>>> completely
> >>>>>>>>>>>> agree
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> view.
> >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> >>>>>> complicated
> >>>>>>>> the
> >>>>>>>>>>> whole
> >>>>>>>>>>>>>> issue.
> >>>>>>>>>>>>>>>>> First
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> >>>> persistence
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> Catalog's
> >>>>>>>>>>>>>>>>>>>> Configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> >>>>>> persistence
> >>>>>>>> for
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> Configuration,
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> >>> to
> >>>> the
> >>>>>>>> Catalog
> >>>>>>>>>>>>>> interface.
> >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> >>> Catalog
> >>>>>>>> instance
> >>>>>>>>>> to a
> >>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> >>>>>>>> implementation
> >>>>>>>>>> will
> >>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> >>>>>>>>>>>>>>>>>>>>>>>>> /**
> >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object.
> >>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object
> >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> >>>>>>>> UnsupportedOperationException if
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> implementing
> >>>>>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>>>>>> not override
> >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> >>> implementation
> >>>> of
> >>>>>> this
> >>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> */
> >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> >>>>>>>> toProperties()
> >>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> >>>>>>>>>>>> UnsupportedOperationException("Please
> >>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> >>>> follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> >>> configured a
> >>>>>>>>>> CatalogStore,
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> >>> registering a
> >>>>>>> Catalog
> >>>>>>>>>>>> instance
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>> catalogName,
> >>>>>>>> Catalog
> >>>>>>>>>>>>> catalog).
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> >>>> Map<String,
> >>>>>>>> String>
> >>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> saved
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> >>>>>> instances
> >>>>>>>> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> >>>>>>>> CatalogStore,
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> >>>>>>> consistency
> >>>>>>>> with
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>> process.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> >>>>>> conflicts
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> >>>>> itself
> >>>>>>> and
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> dedicated
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> >>> loading,
> >>>>> such
> >>>>>>> as
> >>>>>>>>>>>>>>>>> ConfigurationCatalog
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public class
> >>> ConfigurationCatalog
> >>>>>>>> implements
> >>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> ConfigurationCatalog(Map<String,
> >>>>>>> String>
> >>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> >>> the
> >>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> >>> 10:03 AM
> >>>>>> Shammon
> >>>>>>>> FY <
> >>>>>>>>>>>>>>>>>> zjur...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> >>> the
> >>>>>>>> functionality
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> >>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> >>>> multiple
> >>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> >>>>>> implementation
> >>>>>>> in
> >>>>>>>>>> Trino
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> same,
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> >>> only
> >>>>> has
> >>>>>>> one
> >>>>>>>>>>>>>>> `createCatalog`,
> >>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> >>>>>> `CatalogStore`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> >>>>>> another
> >>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> >>> `Map<String,
> >>>>>>> Catalog>
> >>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> >>>> mentioned
> >>>>>>>> before,
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> >>>>>>>> `CatalogManager`.
> >>>>>>>>>> How
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> saving
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> >>> catalogs`
> >>>>> and
> >>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>> catalogStore`
> >>>>>>>>>>>>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> >>> added
> >>>> in
> >>>>>>>>>>>>> `CatalogManager`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>> 5:22 PM
> >>>>> Feng
> >>>>>>>> Jin <
> >>>>>>>>>>>>>>>>>>> jinfeng1...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> >>> I
> >>>> am
> >>>>>> very
> >>>>>>>> sorry
> >>>>>>>>>>> for
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> >>>>>>>>>>>>>>>>>>>>>>>>> caused
> >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> >>>>> original
> >>>>>>>>>>> discussion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> >>> such a
> >>>>>>>> discussion
> >>>>>>>>>>>> before.
> >>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> designing
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> >>> that
> >>>>> the
> >>>>>>>> logic of
> >>>>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>>>> itself
> >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> >>>> cannot
> >>>>>>> only
> >>>>>>>>>>> persist
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> >>>> Catalogs,
> >>>>>>>> because
> >>>>>>>>>> in
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> >>>>> corresponding
> >>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> >>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> >>>>>>> CatalogStore
> >>>>>>>>>>>> interface
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> >>> this
> >>>>>> part, I
> >>>>>>>> also
> >>>>>>>>>>> took
> >>>>>>>>>>>>>>>>> inspiration
> >>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>> handling
> >>>>>>>> catalog
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>> suggestion. I
> >>>>>>> will
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> detailed
> >>>>>>>>>>>>>>>>>>>>>> description
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> >>> part,
> >>>>> and
> >>>>>>> add
> >>>>>>>> it
> >>>>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>> still
> >>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> >>>>>> catalog
> >>>>>>> to
> >>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> >>> very
> >>>>>>> personal
> >>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> >>>>> intuitive
> >>>>>>> to
> >>>>>>>> me.
> >>>>>>>>>> I
> >>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>> see
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> >>>>> statement,
> >>>>>>>> which
> >>>>>>>>>>>> method
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>> `addCatalog`?
> >>>>>>>> I'm
> >>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> >>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> >>>> original
> >>>>>>>> design
> >>>>>>>>>> was
> >>>>>>>>>>> to
> >>>>>>>>>>>>> add
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> directly
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> >>> would
> >>>>>>> disrupt
> >>>>>>>> the
> >>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>> logic.
> >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> >>>>>>>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> >>>>>>>> configuration to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> >>> the
> >>>>> same
> >>>>>>> time
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> >>> neccessary
> >>>> to
> >>>>>>>> provide a
> >>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> >>>>>> otherwise
> >>>>>>> we
> >>>>>>>>>> would
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> introduce
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> >>>> configuration
> >>>>>> for
> >>>>>>>> lazy
> >>>>>>>>>>>>> loading.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>> registration
> >>>>>>>> for
> >>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> >>>>>>> registerCatalog(String
> >>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> >>>>>> interface
> >>>>>>>>>> because
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> >>>>>>>> initialization
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> >>>>>>>> inaccurate. It
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> >>>>>>>> registration. I
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> title
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>> 4:27 PM
> >>>>>>>> Shammon
> >>>>>>>>>> FY <
> >>>>>>>>>>>>>>>>>>> zjur...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> >>>>>>>> `registerCatalog`
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> >>>>> method
> >>>>>>>> named
> >>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> >>> the
> >>>>>> memory
> >>>>>>>>>> catalog
> >>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>> `cache`,
> >>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> >>>>>> exposed
> >>>>>>> to
> >>>>>>>>>>> users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>> Please
> >>>>>>>>>>>>>>>>>>>>>>> correct
> >>>>>>>>>>>>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> >>>>> conclusion
> >>>>>>> of
> >>>>>>>> that
> >>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> >>>>>> interface
> >>>>>>>> and
> >>>>>>>>>>>>> implement
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>>>>> stores
> >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> >>>> external
> >>>>>>>> system.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> >>>>> between
> >>>>>>> the
> >>>>>>>>>>> current
> >>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> >>> of
> >>>>> the
> >>>>>>>>>> discussion
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>>>> [1] ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>>> 3:41 PM
> >>>>>>>> yuxia <
> >>>>>>>>>>>>>>>>>>>>>>> luoyu...@alumni.sjtu.edu.cn>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> >>> driving
> >>>>> this
> >>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>>> handling
> >>>>>>>> catalog
> >>>>>>>>>>> with
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> >>>>> it'll
> >>>>>> be
> >>>>>>>>>> better
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> java
> >>>>>>>>>>>>>>>>>>>>>>>>> doc
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> >>> give a
> >>>>>> brief
> >>>>>>>>>> example
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> >>> name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> >>>> storing
> >>>>>>>> catalog to
> >>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> >>> It's
> >>>>> very
> >>>>>>>> personal
> >>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> >>>>>>> statement,
> >>>>>>>>>> which
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>>>> `addCatalog`?
> >>>>>>>>>> I'm
> >>>>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> >>> SQL
> >>>>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> >>>> neccessary
> >>>>> to
> >>>>>>>>>> provide a
> >>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>>> registration
> >>>>>>>>>> for
> >>>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> >>> catalog
> >>>>>> with
> >>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> >>>>> registration
> >>>>>> is
> >>>>>>>> the
> >>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>> behavior,
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> >>> to
> >>>>>>>> synchronous
> >>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>> before?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> >>>>> `addCatalog`
> >>>>>>> and
> >>>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> >>>>>>>> registration,
> >>>>>>>>>> it
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> >>>> but
> >>>>>> then
> >>>>>>>> the
> >>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>> TABLE
> >>>>>>>>>>>>>>>>>>>>>>>>> statement
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> >>> fail
> >>>>> to
> >>>>>>>> open.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> >>>> change
> >>>>>>> which
> >>>>>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> highlighted
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> >>> part.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> >>>> asynchronous
> >>>>>>>>>>>> registration, I
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> >>>> open
> >>>>>> or
> >>>>>>>>>> register
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> background,
> >>>>>>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> >>> comments,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> >>>> will
> >>>>> be
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> >>>> initialization
> >>>>>>>> process
> >>>>>>>>>>> fails,
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> >>> successfully."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> >>> lazy
> >>>>>>>>>> initialization
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> >>>>>>>>>> jinfeng1...@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> >>>>>>> dev@flink.apache.org
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> >>> 03日
> >>>> 下午
> >>>>>>>> 3:27:45
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> >>>> 295:
> >>>>>>>> Support
> >>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> >>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> >>>> all
> >>>>>> for
> >>>>>>>> your
> >>>>>>>>>>>>> interest
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> >>> catalog
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>> same
> >>>>>>>>>>>>> name
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> >>>> crucial
> >>>>>>>> point.
> >>>>>>>>>>> Based
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> personal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> >>>>>> Map<String,
> >>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> serve
> >>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> >>> catalogs
> >>>>> and
> >>>>>>>> have
> >>>>>>>>>> the
> >>>>>>>>>>>>>> highest
> >>>>>>>>>>>>>>>>>>> priority.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> >>> methods
> >>>>> that
> >>>>>>> can
> >>>>>>>>>> affect
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>> registerCatalog(String
> >>>>>>>>>> catalogName,
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> >>>>> catalog
> >>>>>>>> instance
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>>> unregisterCatalog(String
> >>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>> catalog
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> >>>>> Catalog>
> >>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> >>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> >>>> retrieves
> >>>>>> the
> >>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> >>> Catalog>
> >>>>>>>> catalogs. If
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> exist,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> >>>> corresponding
> >>>>>>>>>>> configuration
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> >>> puts
> >>>>> the
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> >>> methods
> >>>>>> only
> >>>>>>>> modify
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> >>>>>>> catalogName,
> >>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> >>>>>>> properties
> >>>>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> catalogStore
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> >>>>>> catalogName
> >>>>>>>> with
> >>>>>>>>>> the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>> removeCatalog(String
> >>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>>> specified
> >>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> specified
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> >>>>>>> catalogStore.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> >>>> possible
> >>>>>>>> conflict
> >>>>>>>>>>>>>> scenarios:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> >>> in
> >>>> the
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> >>> directly
> >>>>> saved
> >>>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> >>>>>>> Map<String,
> >>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> >>>>> catalog)
> >>>>>>>> method
> >>>>>>>>>>> will
> >>>>>>>>>>>>>> fail.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> >>> the
> >>>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> >>>> method
> >>>>>> can
> >>>>>>>>>> directly
> >>>>>>>>>>>>> save
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> >>>>>>>>>> getCatalog(String
> >>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> >>>> for
> >>>>>>>>>>> initialization
> >>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> >>> already
> >>>>>> exists
> >>>>>>>> in
> >>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> >>>>>>>>>>>>> unregisterCatalog(String
> >>>>>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> >>>>>>>> corresponding to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> >>> confuse
> >>>>>> users
> >>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> >>>>> catalog)`
> >>>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> `registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> >>>> lead
> >>>>>> to
> >>>>>>>>>>>> confusion. I
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> >>> to
> >>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> >>> as
> >>>>>>>> previously
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> >>>>> `registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,Catalog
> >>>>>>>>>>>>>>>>>>>> catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> >>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> >>>> already
> >>>>> an
> >>>>>>>>>> instance,
> >>>>>>>>>>>>>> adding
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> >>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> >>>> necessarily
> >>>>>>>> result in
> >>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>> initialization.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> >>>> about
> >>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> >>>>>>> necessary,
> >>>>>>>> but
> >>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> >>> in
> >>>>> the
> >>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> >>> the
> >>>>>>>>>>> FileCatalogStore
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>> encrypt
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> >>>>> while
> >>>>>>> the
> >>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> require
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> >>> the
> >>>>>>>>>>>>>> `MemoryCatalogStore`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> >>> necessary
> >>>> to
> >>>>>>> have a
> >>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> >>>> saves
> >>>>>>>> Catalog
> >>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> memory.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> >>> to
> >>>>>>>> implement
> >>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> loading
> >>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> >>> introduce
> >>>>>>>> additional
> >>>>>>>>>>> cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> >>>>>>>> registration
> >>>>>>>>>>>> failed?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> >>> critical
> >>>>>>> concern.
> >>>>>>>>>> When
> >>>>>>>>>>>>>>> executing
> >>>>>>>>>>>>>>>>> DDL,
> >>>>>>>>>>>>>>>>>>>> DQL,
> >>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> DML
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> >>>> reference a
> >>>>>>>> specific
> >>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> >>> if it
> >>>>> has
> >>>>>>> not
> >>>>>>>>>> been
> >>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>> yet.
> >>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> >>>>> fails,
> >>>>>>>> these
> >>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> executed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> >>>>>> Catalog>
> >>>>>>>>>> catalogs
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> >>>> have
> >>>>>> the
> >>>>>>>> same
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> >>>> resolved
> >>>>> by
> >>>>>>>>>> referring
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>>>>>>>>> responses.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> >>> the
> >>>>>>> Catalog
> >>>>>>>>>> that
> >>>>>>>>>>>> has
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>> recently
> >>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> >>>> for
> >>>>>>>> subsequent
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> >>>>>> 10:58 PM
> >>>>>>>>>> Xianxun
> >>>>>>>>>>>> Ye
> >>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> >>> this
> >>>>>> Flip,
> >>>>>>>> I do
> >>>>>>>>>>>>> believe
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> Flip
> >>>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> >>>> work
> >>>>>> for
> >>>>>>>> also
> >>>>>>>>>>>>> manages a
> >>>>>>>>>>>>>>> lot
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> catalogs,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> >>>> becomes
> >>>>>>> slow
> >>>>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> loading a
> >>>>>>>>>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> >>> catalogs in
> >>>>> the
> >>>>>>>> user's
> >>>>>>>>>>>> Flink
> >>>>>>>>>>>>>> SQL
> >>>>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>>>>>>> regular
> >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> >>>>> catalogs
> >>>>>>> to
> >>>>>>>>>>> improve
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> speed
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> >>> flip-295, I
> >>>>>> have
> >>>>>>>> some
> >>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> >>>>> catalog
> >>>>>>> with
> >>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>> What
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> >>>>> implement
> >>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> >>> same
> >>>>>>>> question
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> Shammon’s
> >>>>>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>>>>>>>> comment.
> >>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>> catalogs
> >>>>>>> and
> >>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> >>> have
> >>>> the
> >>>>>>> same
> >>>>>>>>>>> catalog
> >>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>> For
> >>>>>>>>>>> example:
> >>>>>>>>>>>>>>>>> catalog1->
> >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> >>> catalog1 ->
> >>>>>>>> hiveCatalog
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> catalogstore.
> >>>>>>>>>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> >>> should
> >>>>>> define
> >>>>>>>>>>> `catalogs`
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> >>> 11:13，Hang
> >>>>>> Ruan
> >>>>>>> <
> >>>>>>>>>>>>>>>>>> ruanhang1...@gmail.com
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 写道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
