Sorry, I forgot to mention I also updated the FLIP to include table apis for connection. It was originally in examples but not in the public api section.
On Tue, Jul 29, 2025 at 10:12 AM Hao Li <h...@confluent.io> wrote: > Hi Leonard, > > Thanks for the feedback and offline sync yesterday. > > > I think connection is a very common need for most catalogs like > MySQLCatalog、KafkaCatalog、HiveCatalog and so on, all of these catalogs need > a connection. > I added `TEMPORARY SYSTEM` connection so it's a global level connection > which can be used for Catalog creation. After syncing with Timo, we propose > to store it first in memory like `TEMPORARY SYSTEM FUNCTION` since this > FLIP is already introducing lots of concepts and interfaces. We can provide > `SYSTEM` connection and related interfaces to persist it in following up > FLIPs. > > > In this case, I think reducing connector development cost is more > important than making is explicit, the connector knows which options is > sensitive or not. > Sure. I updated the FLIP to merge connection options with table options so > it's easier for current connectors. > > > I hope the BasicConnectionFactory can be a common one that can feed most > common users case, otherwise encrypt all options is a good idea. > I updated to `DefaultConnectionFactory` which handles most of the secret > keys. > > Thanks, > Hao > > On Mon, Jul 28, 2025 at 6:13 AM Leonard Xu <xbjt...@gmail.com> wrote: > >> Hey, Hao >> >> Please see my comments as follows: >> >> >> 2. I think we can also modify the create catalog ddl syntax. >> > >> > ``` >> > CREATE CATALOG cat USING CONNECTION mycat.mydb.mysql_prod >> > WITH ( >> > 'type' = 'jdbc' >> > ); >> > ``` >> > >> > Does `mycat.mydb.mysql_prod` exist in another catalog? This feels like a >> > chicken-egg problem. I think for connections to be used for catalog >> > creation, it needs to be system connection similar to system function >> which >> > exist outside of CatalogManager. Maybe we can defer this to later >> > functionality? >> >> I think connection is a very common need for most catalogs like >> MySQLCatalog、KafkaCatalog、HiveCatalog and so on, all of these catalogs need >> a connection. >> >> >> >> 3. It seems the connector factory should merge the with options and >> > connection options together and then create the source/sink. It's >> > better that framework can merge all these options and connectors don't >> need >> > any codes. >> > >> > I think it's better to separate connections with table options and make >> it >> > explicit. Reasons is: the connection here should be a decrypted one. >> It's >> > sensitive and should be handled carefully regarding logging, usage etc. >> > Mixing with original table options makes it hard to do. But the Flip >> used >> > `CatalogConnection` which is an encrypted one. I'll update it to >> > `SensitveConnection`. >> >> （4）Framework-level Resolution : +1 to Shengkai's point about having the >> > framework (DynamicTableFactory) return complete options to reduce >> connector >> > adaptation cost. >> > >> > Please see my explanation for Shengkai's similar question. >> >> In this case, I think reducing connector development cost is more >> important than making is explicit, the connector knows which options is >> sensitive or not. >> >> >> >> 4. Why we need to introduce ConnectionFactory? I think connection is >> like >> > CatalogTable. It should hold the basic information and all information >> in >> > the connection should be stored into secret store. >> > >> > The main reason is to enable user defined connection handling for >> different >> > types. For example, if connection type is `basic`, the corresponding >> > factory can handle basic type secrets (e.g. extract username/password >> from >> > connection options and do encryption). >> > >> >> (2) Configurability of SECRET_FIELDS : Could the hardcoded >> SECRET_FIELDS >> > in BasicConnectionFactory be made configurable (e.g., 'token' vs >> > 'accessKey') for better connector compatibility? >> > >> > This depends on `ConnectionFactory` implementation and can be self >> defined >> > by user. >> >> I hope the BasicConnectionFactory can be a common one that can feed most >> common users case, otherwise encrypt all options is a good idea. >> >> Btw, I also want to push the FLIP forward and start a vote ASAP, thus a >> meeting is welcome if you think it can help finalizing the discussion >> thread. >> >> >> Best, >> Leonard >> >> >> >> > >> >> Hi friends >> >> >> >> I like the updated FLIP goals, that’s what I want. I’ve some feedback: >> >> >> >> （1） Minor: Interface Hierarchy : Why doesn't WritableSecretStore extend >> >> SecretStore? >> >> （2） Configurability of SECRET_FIELDS : Could the hardcoded >> SECRET_FIELDS >> >> in BasicConnectionFactory be made configurable (e.g., 'token' vs >> >> 'accessKey') for better connector compatibility? >> >> （3）Inconsistent Return Types : ConnectionFactory#resolveConnection >> returns >> >> SensitiveConnection, while BasicConnectionFactory#resolveConnection >> returns >> >> Map<String, String>. Should these be aligned? >> >> （4）Framework-level Resolution : +1 to Shengkai's point about having the >> >> framework (DynamicTableFactory) return complete options to reduce >> connector >> >> adaptation cost. >> >> （5）Secret ID Handling : When no encryption is needed, secretId is null >> >> (from secrets.isEmpty() ? null : secretStore.storeSecret(secrets)). >> This >> >> behavior should be explicitly documented in the interfaces. >> >> >> >> Best, >> >> Leonard >> >> >> >>> 2025 7月 24 11:44，Shengkai Fang <fskm...@gmail.com> 写道： >> >>> >> >>> hi. >> >>> >> >>> Sorry for the late reply. I just have some questions: >> >>> >> >>> 1. Why SecretStoreFactory#open throws a CatalogException? I think the >> >>> exteranl system can not handle this exception. >> >>> >> >>> 2. I think we can also modify the create catalog ddl syntax. >> >>> >> >>> ``` >> >>> CREATE CATALOG cat USING CONNECTION mycat.mydb.mysql_prod >> >>> WITH ( >> >>> 'type' = 'jdbc' >> >>> ); >> >>> ``` >> >>> >> >>> 3. It seems the connector factory should merge the with options and >> >>> connection options together and then create the source/sink. It's >> >>> better that framework can merge all these options and connectors don't >> >> need >> >>> any codes. >> >>> >> >>> 4. Why we need to introduce ConnectionFactory? I think connection is >> like >> >>> CatalogTable. It should hold the basic information and all >> information in >> >>> the connection should be stored into secret store. >> >>> >> >>> >> >>> Best, >> >>> Shengkai >> >>> >> >>> >> >>> Timo Walther <twal...@apache.org> 于2025年7月22日周二 22:04写道： >> >>> >> >>>> Hi Mayank, >> >>>> >> >>>> Thanks for updating the FLIP and clearly documenting our discussion. >> >>>> >> >>>> +1 for moving forward with the vote, unless there are objections from >> >>>> others. >> >>>> >> >>>> Cheers, >> >>>> Timo >> >>>> >> >>>> On 22.07.25 02:14, Mayank Juneja wrote: >> >>>>> Hi Ryan and Austin, >> >>>>> >> >>>>> Thanks for your suggestions. I've updated the FLIP with the >> following >> >>>>> additional info - >> >>>>> >> >>>>> 1. *table.secret-store.kind* key to register the SecretStore in a >> yaml >> >>>> file >> >>>>> 2. *updateSecret* method in WritableSecretStore interface >> >>>>> >> >>>>> Thanks, >> >>>>> Mayank >> >>>>> >> >>>>> On Thu, Jul 17, 2025 at 5:42 PM Austin Cawley-Edwards < >> >>>>> austin.caw...@gmail.com> wrote: >> >>>>> >> >>>>>> Hey all, >> >>>>>> >> >>>>>> Thanks for the nice flip all! I’m just reading through – had one >> >>>> question >> >>>>>> on the ALTER CONNECTION implementation flow. Would it make sense >> for >> >> the >> >>>>>> WritableSecretStore to expose a method for updating a secret by >> ID, so >> >>>> it >> >>>>>> can be done atomically? Else, would we need to call delete and >> create >> >>>>>> again, potentially introducing concurrent resolution errors? >> >>>>>> >> >>>>>> Best, >> >>>>>> Austin >> >>>>>> >> >>>>>> On Thu, Jul 17, 2025 at 13:07 Ryan van Huuksloot >> >>>>>> <ryan.vanhuuksl...@shopify.com.invalid> wrote: >> >>>>>> >> >>>>>>> Hi Mayank, >> >>>>>>> >> >>>>>>> Thanks for updating the FLIP. Overall it looks good to me. >> >>>>>>> >> >>>>>>> One question I had related to how someone could choose the >> >> SecretStore >> >>>>>> they >> >>>>>>> want to use if they use something like the SQL Gateway as the >> >>>> entrypoint >> >>>>>> on >> >>>>>>> top of a remote Session cluster. I don't see an explicit way to >> set >> >> the >> >>>>>>> SecretStore in the FLIP. >> >>>>>>> I assume we'll do it similar to the CatalogStore but I wanted to >> call >> >>>>>> this >> >>>>>>> out. >> >>>>>>> >> >>>>>>> table.catalog-store.kind: filetable.catalog-store.file.path: >> >>>>>>> file:///path/to/catalog/store/ >> >>>>>>> >> >>>>>>> Ryan van Huuksloot >> >>>>>>> Staff Engineer, Infrastructure | Streaming Platform >> >>>>>>> [image: Shopify] >> >>>>>>> < >> >>>> >> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Wed, Jul 16, 2025 at 2:22 PM Mayank Juneja < >> >>>> mayankjunej...@gmail.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Hi everyone, >> >>>>>>>> >> >>>>>>>> Thanks for your valuable inputs. I have updated the FLIP with the >> >>>> ideas >> >>>>>>>> proposed earlier in the thread. Looking forward to your feedback. >> >>>>>>>> https://cwiki.apache.org/confluence/x/cYroF >> >>>>>>>> >> >>>>>>>> Best, >> >>>>>>>> Mayank >> >>>>>>>> >> >>>>>>>> On Fri, Jun 27, 2025 at 2:59 AM Leonard Xu <xbjt...@gmail.com> >> >> wrote: >> >>>>>>>> >> >>>>>>>>> Quick response, thanks Mayank, Hao and Timo for the effort. The >> >> new >> >>>>>>>>> proposal looks well, +1 from my side. >> >>>>>>>>> >> >>>>>>>>> Could you draft(update) current FLIP docs thus we can have some >> >>>>>>> specific >> >>>>>>>>> discussions later? >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Best, >> >>>>>>>>> Leonard >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>> 2025 6月 26 15:06，Timo Walther <twal...@apache.org> 写道： >> >>>>>>>>>> >> >>>>>>>>>> Hi everyone, >> >>>>>>>>>> >> >>>>>>>>>> sorry for the late reply, feature freeze kept me busy. Mayank, >> Hao >> >>>>>>> and >> >>>>>>>> I >> >>>>>>>>> synced offline and came up we an improved proposal. Before we >> >> update >> >>>>>>> the >> >>>>>>>>> FLIP let me summarize the most important key facts that >> hopefully >> >>>>>>> address >> >>>>>>>>> most concerns: >> >>>>>>>>>> >> >>>>>>>>>> 1) SecretStore >> >>>>>>>>>> - Similar to CatalogStore, we introduce a SecretStore as the >> >>>>>> highest >> >>>>>>>>> level in TableEnvironment. >> >>>>>>>>>> - SecretStore is initialized with options and potentially >> >>>>>> environment >> >>>>>>>>> variables. Including >> >>>>>> EnvironmentSettings.withSecretStore(SecretStore). >> >>>>>>>>>> - The SecretStore is pluggable and discovered using the regular >> >>>>>>>>> factory-approach. >> >>>>>>>>>> - For example, it could implement Azure Key Vault or other >> cloud >> >>>>>>>>> provider secrets stores. >> >>>>>>>>>> - Goal: Flink and Flink catalogs do not have to deal with >> >> sensitive >> >>>>>>>> data. >> >>>>>>>>>> >> >>>>>>>>>> 2) Connections >> >>>>>>>>>> - Connections are catalog objects identified with 3-part >> >>>>>> identifiers. >> >>>>>>>>> 3-part identifiers are crucial for managability of larger >> projects >> >>>>>> and >> >>>>>>>>> align with existing catalog objects. >> >>>>>>>>>> - They contain connection details, e.g. URL, query parameters, >> and >> >>>>>>>> other >> >>>>>>>>> configuration. >> >>>>>>>>>> - They do not contain secrets, but only pointers to secrets in >> the >> >>>>>>>>> SecretStore. >> >>>>>>>>>> >> >>>>>>>>>> 3) Connection DDL >> >>>>>>>>>> >> >>>>>>>>>> CREATE [TEMPORARY] CONNECTION mycat.mydb.OpenAPI WITH ( >> >>>>>>>>>> 'type' = 'basic' | 'bearer' | 'jwt' | 'oauth' | ..., >> >>>>>>>>>> ... >> >>>>>>>>>> ) >> >>>>>>>>>> >> >>>>>>>>>> - Connection type is pluggable and discovered using the regular >> >>>>>>>>> factory-approach. >> >>>>>>>>>> - The factory extracts secrets and puts them into SecretStore. >> >>>>>>>>>> - The factory only leaves non-confidential options left that >> can >> >> be >> >>>>>>>>> stored in a catalog. >> >>>>>>>>>> >> >>>>>>>>>> When executing: >> >>>>>>>>>> CREATE [TEMPORARY] CONNECTION mycat.mydb.OpenAPI WITH ( >> >>>>>>>>>> 'type' = 'basic', >> >>>>>>>>>> 'url' = 'api.example.com', >> >>>>>>>>>> 'username' = 'bob', >> >>>>>>>>>> 'password' = 'xyz' >> >>>>>>>>>> ) >> >>>>>>>>>> >> >>>>>>>>>> The catalog will receive something similar to: >> >>>>>>>>>> CREATE [TEMPORARY] CONNECTION mycat.mydb.OpenAPI WITH ( >> >>>>>>>>>> 'type' = 'basic', >> >>>>>>>>>> 'url' = 'api.example.com', >> >>>>>>>>>> 'secret.store' = 'azure-key-vault' >> >>>>>>>>>> 'secret.id' = 'secretId' >> >>>>>>>>>> ) >> >>>>>>>>>> >> >>>>>>>>>> - However, the exact property design is up to the connection >> >>>>>> factory. >> >>>>>>>>>> >> >>>>>>>>>> 4) Connection Usage >> >>>>>>>>>> >> >>>>>>>>>> CREATE TABLE t (...) USING CONNECTION mycat.mydb.OpenAPI; >> >>>>>>>>>> >> >>>>>>>>>> - MODEL, FUNCTION, TABLE DDL will support USING CONNECTION >> keyword >> >>>>>>>>> similar to BigQuery. >> >>>>>>>>>> - The connection will be provided in a table/model >> >>>>>> provider/function >> >>>>>>>>> definition factory. >> >>>>>>>>>> >> >>>>>>>>>> 5) CatalogStore / Catalog Initialization >> >>>>>>>>>> >> >>>>>>>>>> Catalog store or catalog can make use of SecretStore to >> retrieve >> >>>>>>>> initial >> >>>>>>>>> credentials for bootstrapping. All objects lower then catalog >> >>>>>>>> store/catalog >> >>>>>>>>> can then use connections. If you think we still need system >> level >> >>>>>>>>> connections, we can support CREATE SYSTEM CONNECTION GlobalName >> >> WITH >> >>>>>>> (..) >> >>>>>>>>> similar to SYSTEM functions directly store in a >> ConnectioManager in >> >>>>>>>>> TableEnvironment. But for now I would suggest to start simple >> with >> >>>>>>>>> per-catalog connections and later evolve the design. >> >>>>>>>>>> >> >>>>>>>>>> Dealing with secrets is a very sensitive topic and I'm clearly >> not >> >>>>>> an >> >>>>>>>>> expert on it. This is why we should try to push the problem to >> >>>>>> existing >> >>>>>>>>> solutions and don't start storing secrets in Flink in any way. >> >> Thus, >> >>>>>>> the >> >>>>>>>>> interfaces will be defined very generic. >> >>>>>>>>>> >> >>>>>>>>>> Looking forward to your feedback. >> >>>>>>>>>> >> >>>>>>>>>> Cheers, >> >>>>>>>>>> Timo >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On 09.06.25 04:01, Leonard Xu wrote: >> >>>>>>>>>>> Thanks Timo for joining this thread. >> >>>>>>>>>>> I agree that this feature is needed by the community; the >> current >> >>>>>>>>> disagreement is only about the implementation method or >> solution. >> >>>>>>>>>>> Your thoughts looks generally good to me, looking forward to >> your >> >>>>>>>>> proposal. >> >>>>>>>>>>> Best, >> >>>>>>>>>>> Leonard >> >>>>>>>>>>>> 2025 6月 6 22:46，Timo Walther <twal...@apache.org> 写道： >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi everyone, >> >>>>>>>>>>>> >> >>>>>>>>>>>> thanks for this healthy discussion. Looking at high number of >> >>>>>>>>> participants, it looks like we definitely want this feature. We >> >> just >> >>>>>>> need >> >>>>>>>>> to figure out the "how". >> >>>>>>>>>>>> >> >>>>>>>>>>>> This reminds me very much of the discussion we had for CREATE >> >>>>>>>>> FUNCTION. There, we discussed whether functions should be named >> >>>>>>> globally >> >>>>>>>> or >> >>>>>>>>> catalog-specific. In the end, we decided for both `CREATE SYSTEM >> >>>>>>>> FUNCTION` >> >>>>>>>>> and `CREATE FUNCTION`, satisfying both the data platform team >> of an >> >>>>>>>>> organization (which might provide system functions) and >> individual >> >>>>>> data >> >>>>>>>>> teams or use cases (scoped by catalog/database). >> >>>>>>>>>>>> >> >>>>>>>>>>>> Looking at other modern vendors like Snowflake there is >> SECRET >> >>>>>>>> (scoped >> >>>>>>>>> to schema) [1] and API INTEGRATION [2] (scoped to account). So >> also >> >>>>>>> other >> >>>>>>>>> vendors offer global and per-team / per-use case connections >> >> details. >> >>>>>>>>>>>> >> >>>>>>>>>>>> In general, I think fitting connections into the existing >> >>>>>> concepts >> >>>>>>>> for >> >>>>>>>>> catalog objects (with three-part identifier) makes managing them >> >>>>>>> easier. >> >>>>>>>>> But I also see the need for global defaults. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Btw keep in mind that a catalog implementation should only >> store >> >>>>>>>>> metadata. Similar how a CatalogTable doesn't store the actual >> >> data, a >> >>>>>>>>> CatalogConnection should not store the credentials. It should >> only >> >>>>>>> offer >> >>>>>>>> a >> >>>>>>>>> factory that allows for storing and retrieving them. In real >> world >> >>>>>>>>> scenarios a factory is most likely backed by a product like >> Azure >> >> Key >> >>>>>>>> Vault. >> >>>>>>>>>>>> >> >>>>>>>>>>>> So code-wise having a ConnectionManager that behaves similar >> to >> >>>>>>>>> FunctionManager sounds reasonable. >> >>>>>>>>>>>> >> >>>>>>>>>>>> +1 for having special syntax instead of using properties. >> This >> >>>>>>> allows >> >>>>>>>>> to access connections in tables, models, functions. And >> catalogs, >> >> if >> >>>>>> we >> >>>>>>>>> agree to have global ones as well. >> >>>>>>>>>>>> >> >>>>>>>>>>>> What do you think? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Let me spend some more thoughts on this and come back with a >> >>>>>>> concrete >> >>>>>>>>> proposal by early next week. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Cheers, >> >>>>>>>>>>>> Timo >> >>>>>>>>>>>> >> >>>>>>>>>>>> [1] >> >>>>>> https://docs.snowflake.com/en/sql-reference/sql/create-secret >> >>>>>>>>>>>> [2] >> >>>>>>>>> >> >>>>>> >> >> https://docs.snowflake.com/en/sql-reference/sql/create-api-integration >> >>>>>>>>>>>> >> >>>>>>>>>>>> On 04.06.25 10:47, Leonard Xu wrote: >> >>>>>>>>>>>>> Hey，Mayank >> >>>>>>>>>>>>> Please see my feedback as following: >> >>>>>>>>>>>>> 1. One of the motivations of this FLIP is to improve >> security. >> >>>>>>>>> However, the current design stores all connection information in >> >> the >> >>>>>>>>> catalog, >> >>>>>>>>>>>>> and each Flink SQL job reads from the catalog during >> >>>>>> compilation. >> >>>>>>>> The >> >>>>>>>>> connection information is passed between SQL Gateway and the >> >>>>>>>>>>>>> catalog in plaintext, which actually introduces new security >> >>>>>>> risks. >> >>>>>>>>>>>>> 2. The name "Connection" should be changed to something like >> >>>>>>>>> ConnectionSpec to clearly indicate that it is a object >> containing >> >>>>>> only >> >>>>>>>>> static >> >>>>>>>>>>>>> properties without a lifecycle. Putting aside the naming >> issue, >> >>>>>> I >> >>>>>>>>> think the current model and hierarchy design is somewhat >> strange. >> >>>>>>> Storing >> >>>>>>>>>>>>> various kinds of connections (e.g., Kafka, MySQL) in the >> same >> >>>>>>>> Catalog >> >>>>>>>>> with hierarchical identifiers like >> >>>>>> catalog-name.db-name.connection-name >> >>>>>>>>>>>>> raises the following questions: >> >>>>>>>>>>>>> (1) What is the purpose of this hierarchical structure of >> >>>>>>> Connection >> >>>>>>>>> object ? >> >>>>>>>>>>>>> (2) If we can use a Connection to create a MySQL table, why >> >>>>>> can't >> >>>>>>> we >> >>>>>>>>> use a Connection to create a MySQL Catalog? >> >>>>>>>>>>>>> 3. Regarding the connector usage examples given in this >> FLIP: >> >>>>>>>>>>>>> ```sql >> >>>>>>>>>>>>> 1 -- Example 2: Using connection for jdbc tables >> >>>>>>>>>>>>> 2 CREATE OR REPLACE CONNECTION mysql_customer_db >> >>>>>>>>>>>>> 3 WITH ( >> >>>>>>>>>>>>> 4 'type' = 'jdbc', >> >>>>>>>>>>>>> 5 'jdbc.url' = 'jdbc:mysql:// >> >>>>>>>>> customer-db.example.com:3306/customerdb', >> >>>>>>>>>>>>> 6 'jdbc.connection.ssl.enabled' = 'true' >> >>>>>>>>>>>>> 7 ); >> >>>>>>>>>>>>> 8 >> >>>>>>>>>>>>> 9 CREATE TABLE customers ( >> >>>>>>>>>>>>> 10 customer_id INT, >> >>>>>>>>>>>>> 11 PRIMARY KEY (customer_id) NOT ENFORCED >> >>>>>>>>>>>>> 12 ) WITH ( >> >>>>>>>>>>>>> 13 'connector' = 'jdbc', >> >>>>>>>>>>>>> 14 'jdbc.connection' = 'mysql_customer_db', >> >>>>>>>>>>>>> 15 'jdbc.connection.ssl.enabled' = 'true', >> >>>>>>>>>>>>> 16 'jdbc.connection.max-retry-timeout' = '60s', >> >>>>>>>>>>>>> 17 'jdbc.table-name' = 'customers', >> >>>>>>>>>>>>> 18 'jdbc.lookup.cache' = 'PARTIAL' >> >>>>>>>>>>>>> 19 ); >> >>>>>>>>>>>>> ``` >> >>>>>>>>>>>>> I see three issues from SQL semantics and Connector >> >>>>>> compatibility >> >>>>>>>>> perspectives: >> >>>>>>>>>>>>> (1) Look at line 14: `mysql_customer_db` is an object >> >> identifier >> >>>>>>> of >> >>>>>>>> a >> >>>>>>>>> CONNECTION defined in SQL. However, this identifier is >> referenced >> >>>>>>>>>>>>> via a string value inside the table’s WITH clause, which >> >>>>>> feel >> >>>>>>>>> hack for me. >> >>>>>>>>>>>>> (2) Look at lines 14–16: the use of the specific prefix >> >>>>>>>>> `jdbc.connection` will confuse users because `connection.xx` >> maybe >> >>>>>>>> already >> >>>>>>>>> used as >> >>>>>>>>>>>>> a prefix for existing configuration items. >> >>>>>>>>>>>>> (3) Look at lines 14–18: Why do all existing configuration >> >>>>>> options >> >>>>>>>>> need to be prefixed with `jdbc`, even they’re not related to >> >>>>>> Connection >> >>>>>>>>> properties? >> >>>>>>>>>>>>> This completely changes user habits — is it backward >> >> compatible? >> >>>>>>>>>>>>> In my opinion, Connection should be a model independent of >> >> both >> >>>>>>>>> Catalog and Table, and can be referenced by all >> >>>>>> catalog/table/udf/model >> >>>>>>>>> object. >> >>>>>>>>>>>>> It should be managed by a Component such as a >> ConnectionManager >> >>>>>> to >> >>>>>>>>> enable reuse. For security purposes, authentication mechanisms >> >> could >> >>>>>>>>>>>>> be supported within the ConnectionManager. >> >>>>>>>>>>>>> Best, >> >>>>>>>>>>>>> Leonard >> >>>>>>>>>>>>>> 2025 6月 4 02:04，Martijn Visser <martijnvis...@apache.org> >> 写道： >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi all, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> First of all, I think having a Connection resource is >> >> something >> >>>>>>>> that >> >>>>>>>>> will >> >>>>>>>>>>>>>> be beneficial for Apache Flink. I could see that being >> >> extended >> >>>>>>> in >> >>>>>>>>> the >> >>>>>>>>>>>>>> future to allow for easier secret handling [1]. >> >>>>>>>>>>>>>> In my mental mind, I'm comparing this proposal against >> SQL/MED >> >>>>>>> from >> >>>>>>>>> the ISO >> >>>>>>>>>>>>>> standard [2]. I do think that SQL/MED isn't a very user >> >>>>>> friendly >> >>>>>>>>> syntax >> >>>>>>>>>>>>>> though, looking at Postgres for example [3]. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I think it's a valid question if Connection should be >> >>>>>> considered >> >>>>>>>>> with a >> >>>>>>>>>>>>>> catalog or database-level scope. @Ryan can you share >> something >> >>>>>>>> more, >> >>>>>>>>> since >> >>>>>>>>>>>>>> you've mentioned "Note: I much prefer catalogs for this >> case. >> >>>>>>> Which >> >>>>>>>>> is what >> >>>>>>>>>>>>>> we use internally to manage connection properties". It >> looks >> >>>>>> like >> >>>>>>>>> there >> >>>>>>>>>>>>>> isn't a strong favourable approach looking at other vendors >> >>>>>>> (like, >> >>>>>>>>>>>>>> Databricks does scopes it on a Unity catalog, Snowflake on >> a >> >>>>>>>> database >> >>>>>>>>>>>>>> level). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Also looking forward to Leonard's input. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Best regards, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Martijn >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-36818 >> >>>>>>>>>>>>>> [2] https://www.iso.org/standard/84804.html >> >>>>>>>>>>>>>> [3] >> >>>>>>> https://www.postgresql.org/docs/current/sql-createserver.html >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Fri, May 30, 2025 at 5:07 AM Leonard Xu < >> xbjt...@gmail.com >> >>> >> >>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hey Mayank. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks for the FLIP, I went through this FLIP quickly and >> >>>>>> found >> >>>>>>>> some >> >>>>>>>>>>>>>>> issues which I think we >> >>>>>>>>>>>>>>> need to deep discuss later. As we’re on a short Dragon >> boat >> >>>>>>>>> Festival, >> >>>>>>>>>>>>>>> could you kindly hold >> >>>>>>>>>>>>>>> on this thread? and we will back to continue the FLIP >> >> discuss. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>> Leonard >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> 2025 4月 29 23:07，Mayank Juneja <mayankjunej...@gmail.com >> > >> >>>>>> 写道： >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Hi all, >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> I would like to open up for discussion a new FLIP-529 >> [1]. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Motivation: >> >>>>>>>>>>>>>>>> Currently, Flink SQL handles external connectivity by >> >>>>>> defining >> >>>>>>>>> endpoints >> >>>>>>>>>>>>>>>> and credentials in table configuration. This approach >> >>>>>> prevents >> >>>>>>>>>>>>>>> reusability >> >>>>>>>>>>>>>>>> of these connections and makes table definition less >> secure >> >>>>>> by >> >>>>>>>>> exposing >> >>>>>>>>>>>>>>>> sensitive information. >> >>>>>>>>>>>>>>>> We propose the introduction of a new "connection" >> resource >> >> in >> >>>>>>>>> Flink. This >> >>>>>>>>>>>>>>>> will be a pluggable resource configured with a remote >> >>>>>> endpoint >> >>>>>>>> and >> >>>>>>>>>>>>>>>> associated access key. Once defined, connections can be >> >>>>>> reused >> >>>>>>>>> across >> >>>>>>>>>>>>>>> table >> >>>>>>>>>>>>>>>> definitions, and eventually for model definition (as >> >>>>>> discussed >> >>>>>>> in >> >>>>>>>>>>>>>>> FLIP-437) >> >>>>>>>>>>>>>>>> for inference, enabling seamless and secure integration >> with >> >>>>>>>>> external >> >>>>>>>>>>>>>>>> systems. >> >>>>>>>>>>>>>>>> The connection resource will provide a new, optional way >> to >> >>>>>>>> manage >> >>>>>>>>>>>>>>> external >> >>>>>>>>>>>>>>>> connectivity in Flink. Existing methods for table >> >> definitions >> >>>>>>>> will >> >>>>>>>>> remain >> >>>>>>>>>>>>>>>> unchanged. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/x/cYroF >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Best Regards, >> >>>>>>>>>>>>>>>> Mayank Juneja >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> *Mayank Juneja* >> >>>>>>>> Product Manager | Data Streaming and AI >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >> >> >> >> >>
