Hi Hao,
> System connection is
not bound to catalog and can be used to create catalog.
It looks good to me.
Best,
Hongshun
> 2025年8月7日 00:05，Hao Li 写道：
>
> System connection is
> not bound to catalog and can be used to create catalog.
Hi Hongshun,
Thanks for your question. Answer your question below:
> Does this mean that users are still required to provide initial
credentials (e.g., username and password) when creating the catalog?
There are two types of connection: system and catalog. System connection is
not bound to catal
Hi Mayank,
Sorry for joining this FLIP discussion late. I can see that it significantly
reduces complexity and improves security when creating tables.
However, I have a question regarding its usage in the context of catalogs:
In the current FLIP, when creating a connection within a catalog, we u
> I suggest we modify ObjectIdentifier and let it have either 3 or 1 part. Then
> we can also remove the need for FunctionIdentifier which exists for exactly
> this purpose.
+1 from my side
Best,
Leonard
Hi Timo,
Thanks for the suggestion. I updated the FLIP with `ObjectIdentifier`
taking only objectName and made it clear that the system connection uses
only a simple identifier instead of compound identifier.
Thanks,
Hao
On Thu, Jul 31, 2025 at 6:12 AM Timo Walther wrote:
> Thanks for updating
Thanks for updating the FLIP and starting a VOTE thread. I have two last
minute questions before I cast my vote:
It seems the definition of TEMPORARY SYSTEM CONNECTION is not precise
enough. In my point of view, system connections are single part
identifiers (similar to functions). And catalog
Thanks Hao for the effort to push the FLIP forward, I believe current design
would satisfy most user cases.
+1 to start a vote.
Best,
Leonard
> 2025 7月 30 02:27，Hao Li 写道：
>
> Sorry, I forgot to mention I also updated the FLIP to include table apis
> for connection. It was originally in exa
Sorry, I forgot to mention I also updated the FLIP to include table apis
for connection. It was originally in examples but not in the public api
section.
On Tue, Jul 29, 2025 at 10:12 AM Hao Li wrote:
> Hi Leonard,
>
> Thanks for the feedback and offline sync yesterday.
>
> > I think connection
Hi Leonard,
Thanks for the feedback and offline sync yesterday.
> I think connection is a very common need for most catalogs like
MySQLCatalog、KafkaCatalog、HiveCatalog and so on, all of these catalogs need
a connection.
I added `TEMPORARY SYSTEM` connection so it's a global level connection
which
Hey, Hao
Please see my comments as follows:
>> 2. I think we can also modify the create catalog ddl syntax.
>
> ```
> CREATE CATALOG cat USING CONNECTION mycat.mydb.mysql_prod
> WITH (
>'type' = 'jdbc'
> );
> ```
>
> Does `mycat.mydb.mysql_prod` exist in another catalog? This feels like a
>
Hi Yanquan,
Thanks for the feedback.
> (1) I agree with the design of SecretStore. I hope to add the results of
the 'SHOW Connections' query to the document, 'secret.id' should be
included in the results, This helps us establish the association between
tables and connections.
I think we can outp
Hi Mayank and Hao,
Thanks for the updates and comments. Here are some of my opinions:
(1) I agree with the design of SecretStore. I hope to add the results of
the 'SHOW Connections' query to the document, 'secret.id' should be
included in the results, This helps us establish the association betwee
Hi Shengkai, Leonard,
Thanks for the feedback.
For Shengkai's feedback:
> 1. Why SecretStoreFactory#open throws a CatalogException? I think the
exteranl system can not handle this exception.
You are right, we can make it throw `Exception`
> 2. I think we can also modify the create catalog ddl s
Hi friends
I like the updated FLIP goals, that’s what I want. I’ve some feedback:
（1） Minor: Interface Hierarchy : Why doesn't WritableSecretStore extend
SecretStore?
（2） Configurability of SECRET_FIELDS : Could the hardcoded SECRET_FIELDS in
BasicConnectionFactory be made configurable (e.g., '
hi.
Sorry for the late reply. I just have some questions:
1. Why SecretStoreFactory#open throws a CatalogException? I think the
exteranl system can not handle this exception.
2. I think we can also modify the create catalog ddl syntax.
```
CREATE CATALOG cat USING CONNECTION mycat.mydb.mysql_pr
Hi Mayank,
Thanks for updating the FLIP and clearly documenting our discussion.
+1 for moving forward with the vote, unless there are objections from
others.
Cheers,
Timo
On 22.07.25 02:14, Mayank Juneja wrote:
Hi Ryan and Austin,
Thanks for your suggestions. I've updated the FLIP with the
Hi Ryan and Austin,
Thanks for your suggestions. I've updated the FLIP with the following
additional info -
1. *table.secret-store.kind* key to register the SecretStore in a yaml file
2. *updateSecret* method in WritableSecretStore interface
Thanks,
Mayank
On Thu, Jul 17, 2025 at 5:42 PM Austin
Hey all,
Thanks for the nice flip all! I’m just reading through – had one question
on the ALTER CONNECTION implementation flow. Would it make sense for the
WritableSecretStore to expose a method for updating a secret by ID, so it
can be done atomically? Else, would we need to call delete and creat
Hi Mayank,
Thanks for updating the FLIP. Overall it looks good to me.
One question I had related to how someone could choose the SecretStore they
want to use if they use something like the SQL Gateway as the entrypoint on
top of a remote Session cluster. I don't see an explicit way to set the
Sec
Hi everyone,
Thanks for your valuable inputs. I have updated the FLIP with the ideas
proposed earlier in the thread. Looking forward to your feedback.
https://cwiki.apache.org/confluence/x/cYroF
Best,
Mayank
On Fri, Jun 27, 2025 at 2:59 AM Leonard Xu wrote:
> Quick response, thanks Mayank, Hao
Quick response, thanks Mayank, Hao and Timo for the effort. The new proposal
looks well, +1 from my side.
Could you draft(update) current FLIP docs thus we can have some specific
discussions later?
Best,
Leonard
> 2025 6月 26 15:06，Timo Walther 写道：
>
> Hi everyone,
>
> sorry for the late
Hi everyone,
Thanks Timo + company, for the updated proposal. From my perspective, the
systems and syntaxes proposed are much better. It still allows for the
functionality required but has better segregation of duties.Would love to
see an updated FLIP.
Would it be worth splitting the proposal int
Hi everyone,
sorry for the late reply, feature freeze kept me busy. Mayank, Hao and I
synced offline and came up we an improved proposal. Before we update the
FLIP let me summarize the most important key facts that hopefully
address most concerns:
1) SecretStore
- Similar to CatalogStore, we
Thanks Timo for joining this thread.
I agree that this feature is needed by the community; the current disagreement
is only about the implementation method or solution.
Your thoughts looks generally good to me, looking forward to your proposal.
Best,
Leonard
> 2025 6月 6 22:46，Timo Walther 写
Hi everyone,
thanks for this healthy discussion. Looking at high number of
participants, it looks like we definitely want this feature. We just
need to figure out the "how".
This reminds me very much of the discussion we had for CREATE FUNCTION.
There, we discussed whether functions should b
Hey，Mayank
Please see my feedback as following:
1. One of the motivations of this FLIP is to improve security. However, the
current design stores all connection information in the catalog,
and each Flink SQL job reads from the catalog during compilation. The
connection information is passed be
Thanks for Mayank's proposal!
My bad! I had the wrong person listed as the initiator.
Best,
Shengkai
Shengkai Fang 于2025年6月4日周三 12:07写道：
> Hi, LiHao.
>
> Thanks for the proposal. The idea of connection is very attractive and
> really helps Flink users to manage their secret. But I have some co
Hi, LiHao.
Thanks for the proposal. The idea of connection is very attractive and
really helps Flink users to manage their secret. But I have some concerns
about this FLIP:
*Proposal for a Top-Level Connection Concept in Flink SQL*
I believe connection should be elevated as a core, top-level conc
Hi all,
First of all, I think having a Connection resource is something that will
be beneficial for Apache Flink. I could see that being extended in the
future to allow for easier secret handling [1].
In my mental mind, I'm comparing this proposal against SQL/MED from the ISO
standard [2]. I do th
Hey Mayank.
Thanks for the FLIP, I went through this FLIP quickly and found some issues
which I think we
need to deep discuss later. As we’re on a short Dragon boat Festival, could
you kindly hold
on this thread? and we will back to continue the FLIP discuss.
Best,
Leonard
> 2025 4月 29 23
Hi Mayank,
Thanks for the response. I have a few more questions.
1. Why not use a catalog that has default properties? That way you could
re-use properties if the object exists in that catalog/database.
---
If we are insistent on Connections.
1. I don't understand the distinction between a "CO
Thanks all for the feedback on the FLIP. If there are no more discussions,
I will start voting on this tomorrow.
On Wed, May 21, 2025 at 6:00 AM Mayank Juneja
wrote:
> Hi Ryan,
>
> Thanks for the feedback.
>
> Just a quick clarification—DESCRIBE TABLE in Flink does *not* show table
> properties;
Hi Ryan,
Thanks for the feedback.
Just a quick clarification—DESCRIBE TABLE in Flink does *not* show table
properties; I believe you might be thinking of SHOW CREATE TABLE, which
does include properties as part of the DDL output. The goal of SHOW CREATE
TABLE is to provide a fully reproducible st
Hi Mayank,
Overall I think the idea of having shared properties (secrets or otherwise)
at the database layer is really interesting/useful. Even outside of the ML
scope.
However, I have a concern:
When I describe a table, I would not like it if all properties did not
appear in the returned results
Hi Mayank,
Thanks for the initiative! I think this functionality would be a really
great addition to Flink.
+1 for the proposal.
Thanks,
Yash
On Tue, May 6, 2025 at 9:36 AM Mayank Juneja
wrote:
> Hi Ferenc,
>
> Thanks for your question! We agree that a CONNECTION could logically be a
> catalo
Hi Ferenc,
Thanks for your question! We agree that a CONNECTION could logically be a
catalog-level resource, especially since it’s intended to be reused across
multiple tables. However, we think there’s value in defining it at the
database level to introduce a layer of isolation and scoping.
This
Thanks Mayank for the proposal. I think it's a great addition to Flink to
define secure connectivity in general for table, model and other resources
later on. +1.
Hao
On Fri, May 2, 2025 at 5:32 AM Gustavo de Morais
wrote:
> Hi Mayank,
>
> Thanks for the initiative. Looking at the FLIP, this lo
Hi Mayank,
Thanks for the initiative. Looking at the FLIP, this looks like a
well-thought-out proposal that addresses a clear need for more secure and
reusable external connections in Flink SQL and Table API. Separating
connection details would a valuable improvement.
Best Regards,
Gustavo
Am Fr
Hi Mayank,
Thank you for starting the discussion! In general, I think such functionality
would be a really great addition to Flink.
Could you pls. elaborate a bit more one what is the reason of defining a
`connection` resource on the database level instead of the catalog level?
If I think about `
Hi all,
I would like to open up for discussion a new FLIP-529 [1].
Motivation:
Currently, Flink SQL handles external connectivity by defining endpoints
and credentials in table configuration. This approach prevents reusability
of these connections and makes table definition less secure by exposin
40 matches
Mail list logo
