Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[yu zelin]

Congratulations!

Best,
Yu Zelin

> 2023年10月16日 09:56，Jark Wu  写道：
> 
> Hi, everyone
> 
> On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
> Committer.
> 
> Ron has been continuously contributing to the Flink project for many years,
> authored and reviewed a lot of codes. He mainly works on Flink SQL parts
> and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
> knowledge of the Batch SQL and improved a lot of batch performance in the
> past several releases. He is also quite active in mailing lists,
> participating in discussions and answering user questions.
> 
> Please join me in congratulating Ron Liu for becoming a Flink Committer!
> 
> Best,
> Jark Wu (on behalf of the Flink PMC)

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[yu zelin]

Congratulations!

Best,
Yu Zelin

> 2023年10月16日 09:58，Jark Wu  写道：
> 
> Hi, everyone
> 
> On behalf of the PMC, I'm very happy to announce Jane Chan as a new Flink
> Committer.
> 
> Jane started code contribution in Jan 2021 and has been active in the Flink
> community since. She authored more than 60 PRs and reviewed more than 40
> PRs. Her contribution mainly revolves around Flink SQL, including Plan
> Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER TABLE
> statements (FLINK-21634). Jane participated deeply in development
> discussions and also helped answer user question emails. Jane was also a
> core contributor of Flink Table Store (now Paimon) when the project was in
> the early days.
> 
> Please join me in congratulating Jane Chan for becoming a Flink Committer!
> 
> Best,
> Jark Wu (on behalf of the Flink PMC)

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[yu zelin]

Congratulations!

Best,
Yu Zelin

> 2023年3月27日 17:23，Yu Li  写道：
> 
> Dear Flinkers,
> 
> As you may have noticed, we are pleased to announce that Flink Table Store 
> has joined the Apache Incubator as a separate project called Apache 
> Paimon(incubating) [1] [2] [3]. The new project still aims at building a 
> streaming data lake platform for high-speed data ingestion, change data 
> tracking and efficient real-time analytics, with the vision of supporting a 
> larger ecosystem and establishing a vibrant and neutral open source community.
> 
> We would like to thank everyone for their great support and efforts for the 
> Flink Table Store project, and warmly welcome everyone to join the 
> development and activities of the new project. Apache Flink will continue to 
> be one of the first-class citizens supported by Paimon, and we believe that 
> the Flink and Paimon communities will maintain close cooperation.
> 
> 亲爱的Flinkers，
> 
> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache 孵化器独立孵化 [1] [2] 
> [3]。新项目的名字是 Apache 
> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
> 
> 在这里我们要感谢大家对 Flink Table Store 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink 
> 将继续作为 Paimon 支持的主力计算引擎之一，我们也相信 Flink 和 Paimon 社区将继续保持密切合作。
> 
> Best Regards,
> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
> 
> 致礼，
> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
> 
> [1] https://paimon.apache.org/
> [2] https://github.com/apache/incubator-paimon
> [3] https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal

[RESULT][VOTE] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, all,

FLIP-275: Support Remote SQL Client Based on SQL Gateway[1] has been accepted.

There are 3 bindings, and 2 non-bindings as follows:

ShengKai Fang (binding),
Paul Lam (non-binding),
Hang Ruan (non-binding)
Godfrey He (binding)
Jark Wu (binding)

There are no votes against it.

Best,
Yu Zelin

[1] https://cwiki.apache.org/confluence/x/T48ODg

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi all,

Recently I have received some feedbacks about the REST Endpoint modification. 
The main point
is use ‘ResultSet’ as a part of FetchResultsResponseBody’ is not convenient for 
serialization and
deserialization. So I think it’s better to introduce a new ‘ResultInfo’ to 
carry the data. The ‘ResultInfo’
will carry the row format information and be serialized and deserialized 
according to the row format.

In FLIP, I have modified the Section: Public Interface -> REST Endpoint 
Modification. The main 
change is the ‘FetchJsonFormatResultsResponseBody' and 
‘FetchPlainTextResultsResponseBody’
was deleted and the ‘overview of fetching results REST API’ was minor modified. 

Best,
Yu Zelin

> 2022年12月13日 17:13，yu zelin  写道：
> 
> Hi everyone,
> 
> Sorry for the incorrect message in my last email. I want to start the vote on 
> Wednesday 
> as long as there are no questions in this period.
> 
> Best,
> Yu Zelin
> 
> On Tue, Dec 13, 2022 at 5:08 PM yu zelin  <mailto:yuzelin@gmail.com>> wrote:
>> Hi, everyone,
>> 
>> Looks like our new design is similar to Timo’s suggestion, and considering 
>> that there has
>> no response from other devs for a long time, I want to start the vote on 
>> Thursday.  
>> 
>> 
>> Best,
>> Yu Zelin
>> 
>>> 2022年12月13日 16:23，yu zelin >> <mailto:yuzelin@gmail.com>> 写道：
>>> 
>>> Hi, Timo,
>>> 
>>> Thanks for your suggestion. Recently I have discussed with @Godfrey He, 
>>> @Shengkai Fang 
>>> and @Jark Wu about the `RowFormat` (Thanks for all your suggestions). We 
>>> finally came to 
>>> a consensus which is similar to your suggestion. The details are as follows:
>>> 
>>> 1. Add a REST query parameter ‘RowFormat’ = JSON/PLAIN_TEXT to tell the 
>>> REST Endpoint
>>> how to deserialize the RowData int ResultSet.
>>> 
>>> JSON format means the RowData will be serialized to JSON format, which 
>>> contains original 
>>> LogicalType information, so it can be deserialized back to RowData.
>>> 
>>> PLAIN_TEXT format means the RowData will be serialized to 
>>> SQL-compliant, plain strings. 
>>> The SQL Client can print the strings directly.
>>> 
>>> The example URI for fetching results is:
>>> > /v2/sessions/:session_handle/operations/:operation_handle/result/:token?rowFormat=PLAIN_TEXT
>>> 
>>> 2. Introduce two response bodies for fetching results in two formats.
>>> 
>>> For more details, please take a look at the FLIP 
>>> [https://cwiki.apache.org/confluence/x/T48ODg]. 
>>> I have updated it with an example of query response bodies in two format in 
>>> section:
>>> Public Interface -> REST Endpoint Modification.
>>> 
>>>> 2022年12月12日 18:09，Timo Walther >>> <mailto:twal...@apache.org>> 写道：
>>>> 
>>>> Hi everyone,
>>>> 
>>>> sorry to jump into this discussion so late.
>>>> 
>>>> > So we decided to revert the RowFormat related changes and let the client 
>>>> > to resolve the print format.
>>>> 
>>>> Could you elaborate a bit on this topic in the FLIP? I still believe that 
>>>> we need 2 types of output formats.
>>>> 
>>>> Format A: for the SQL Client CLI and other interactive notebooks that just 
>>>> uses SQL CAST(... AS STRING) semantics executed on the server side
>>>> 
>>>> Format B: for JDBC SDK or other machine-readable downstream libraries
>>>> 
>>>> Take a TIMESTAMP WITH LOCAL TIME ZONE as an example. The string 
>>>> representation depends on a session configuration option. Clients might 
>>>> not be aware of this session option, so the formatting must happen on the 
>>>> server side.
>>>> 
>>>> However, when the downstream consumer is a library, maybe the library 
>>>> would like to get the raw millis/nanos since epoch.
>>>> 
>>>> Also nested rows and collections might be better encoded with format B for 
>>>> libraries but interactive sessions are happy if nested types are already 
>>>> formatted server-side, so not every client needs custom code for the 
>>>> formatting.
>>>> 
>>>> Regards,
>>>> Timo
>>>> 
>>>> 
>>>> 
>>>> On 06.12.22 15:13, godfrey he wrote:
>>>>> Hi, zeklin
>>>>>> The CLI will use default print style for the non-quer

[VOTE] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, all,

Thanks for all your feedbacks so far. Through the discussion on this 
thread[1], I think we have came to a consensus, so I’d like to start a 
vote on FLIP-275[2].

The vote will last for at least 72 hours (Dec 19th, 13:00 GMT, excluding 
weekend days) unless there is an objection or insufficient vote.

Best,
Yu Zelin

[1] https://lists.apache.org/thread/zpx64l0z91b0sz0scv77h0g13ptj4xxo 
[2] https://cwiki.apache.org/confluence/x/T48ODg

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi everyone,

Sorry for the incorrect message in my last email. I want to start the vote
on Wednesday
as long as there are no questions in this period.

Best,
Yu Zelin

On Tue, Dec 13, 2022 at 5:08 PM yu zelin  wrote:

> Hi, everyone,
>
> Looks like our new design is similar to Timo’s suggestion, and considering
> that there has
> no response from other devs for a long time, I want to start the vote on
> Thursday.
>
>
> Best,
> Yu Zelin
>
> 2022年12月13日 16:23，yu zelin  写道：
>
> Hi, Timo,
>
> Thanks for your suggestion. Recently I have discussed with @Godfrey He,
> @Shengkai Fang
> and @Jark Wu about the `RowFormat` (Thanks for all your suggestions). We
> finally came to
> a consensus which is similar to your suggestion. The details are as
> follows:
>
> 1. Add a REST query parameter ‘*RowFormat*’ = JSON/PLAIN_TEXT to tell the
> REST Endpoint
> how to deserialize the RowData int ResultSet.
>
> JSON format means the RowData will be serialized to JSON format, which
> contains original
> *LogicalType* information, so it can be deserialized back to RowData.
>
> PLAIN_TEXT format means the RowData will be serialized to
> SQL-compliant, plain strings.
> The SQL Client can print the strings directly.
>
> The example URI for fetching results is:
> > /v2/sessions/:session_handle/operations/:operation_handle/result/:token?
> rowFormat=PLAIN_TEXT
>
> 2. Introduce two response bodies for fetching results in two formats.
>
> For more details, please take a look at the FLIP [
> https://cwiki.apache.org/confluence/x/T48ODg].
> I have updated it with an example of query response bodies in two format
> in section:
> Public Interface -> REST Endpoint Modification.
>
> 2022年12月12日 18:09，Timo Walther  写道：
>
> Hi everyone,
>
> sorry to jump into this discussion so late.
>
> > So we decided to revert the RowFormat related changes and let the client
> to resolve the print format.
>
> Could you elaborate a bit on this topic in the FLIP? I still believe that
> we need 2 types of output formats.
>
> Format A: for the SQL Client CLI and other interactive notebooks that just
> uses SQL CAST(... AS STRING) semantics executed on the server side
>
> Format B: for JDBC SDK or other machine-readable downstream libraries
>
> Take a TIMESTAMP WITH LOCAL TIME ZONE as an example. The string
> representation depends on a session configuration option. Clients might not
> be aware of this session option, so the formatting must happen on the
> server side.
>
> However, when the downstream consumer is a library, maybe the library
> would like to get the raw millis/nanos since epoch.
>
> Also nested rows and collections might be better encoded with format B for
> libraries but interactive sessions are happy if nested types are already
> formatted server-side, so not every client needs custom code for the
> formatting.
>
> Regards,
> Timo
>
>
>
> On 06.12.22 15:13, godfrey he wrote:
>
> Hi, zeklin
>
> The CLI will use default print style for the non-query result.
>
> Please make sure the print results of EXPLAIN/DESC/SHOW CREATE TABLE
> commands are clear.
>
> We think it’s better to add the root cause to the ErrorResponseBody.
>
> LGTM
> Best,
> Godfrey
> yu zelin  于2022年12月6日周二 17:51写道：
>
>
> Hi, Godfrey
>
> Thanks for your feedback. Below is my thoughts about your questions.
>
> 1. About RowFormat.
> I agree to your opinion. So we decided to revert the RowFormat related
> changes
> and let the client to resolve the print format.
>
> 2. About ContentType
> I agree that the definition of the ContentType is not clear. But how to
> define the
> statement type is another big question. So, we decided to only tell the
> query result
> and non-query result apart. The CLI will use default print style for the
> non-query
> result.
>
> 3. About ErrorHandling
> I think reuse the current ErrorResponseBody is good, but parse the root
> cause
> from the exception stack strings is quite hacking. We think it’s better to
> add the
> root cause to the ErrorResponseBody.
>
> 4. About Runtime REST API Modifications
> I agree, too. This part is moved to the ‘Future Work’.
>
> Best,
> Yu Zelin
>
>
> 2022年12月5日 18:33，godfrey he  写道：
>
> Hi Zelin,
>
> Thanks for driving this discussion.
>
> I have a few comments,
>
> Add RowFormat to ResultSet to indicate the format of rows.
>
> We should not require SqlGateway server to meet the display
> requirements of a CliClient.
> Because different CliClients may have different display style. The
> server just need to response the data,
> and the CliClient prints the result as 

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, everyone,

Looks like our new design is similar to Timo’s suggestion, and considering that 
there has
no response from other devs for a long time, I want to start the vote on 
Thursday.  


Best,
Yu Zelin

> 2022年12月13日 16:23，yu zelin  写道：
> 
> Hi, Timo,
> 
> Thanks for your suggestion. Recently I have discussed with @Godfrey He, 
> @Shengkai Fang 
> and @Jark Wu about the `RowFormat` (Thanks for all your suggestions). We 
> finally came to 
> a consensus which is similar to your suggestion. The details are as follows:
> 
> 1. Add a REST query parameter ‘RowFormat’ = JSON/PLAIN_TEXT to tell the REST 
> Endpoint
> how to deserialize the RowData int ResultSet.
> 
> JSON format means the RowData will be serialized to JSON format, which 
> contains original 
> LogicalType information, so it can be deserialized back to RowData.
> 
> PLAIN_TEXT format means the RowData will be serialized to SQL-compliant, 
> plain strings. 
> The SQL Client can print the strings directly.
> 
> The example URI for fetching results is:
> > /v2/sessions/:session_handle/operations/:operation_handle/result/:token?rowFormat=PLAIN_TEXT
> 
> 2. Introduce two response bodies for fetching results in two formats.
> 
> For more details, please take a look at the FLIP 
> [https://cwiki.apache.org/confluence/x/T48ODg]. 
> I have updated it with an example of query response bodies in two format in 
> section:
> Public Interface -> REST Endpoint Modification.
> 
>> 2022年12月12日 18:09，Timo Walther  写道：
>> 
>> Hi everyone,
>> 
>> sorry to jump into this discussion so late.
>> 
>> > So we decided to revert the RowFormat related changes and let the client 
>> > to resolve the print format.
>> 
>> Could you elaborate a bit on this topic in the FLIP? I still believe that we 
>> need 2 types of output formats.
>> 
>> Format A: for the SQL Client CLI and other interactive notebooks that just 
>> uses SQL CAST(... AS STRING) semantics executed on the server side
>> 
>> Format B: for JDBC SDK or other machine-readable downstream libraries
>> 
>> Take a TIMESTAMP WITH LOCAL TIME ZONE as an example. The string 
>> representation depends on a session configuration option. Clients might not 
>> be aware of this session option, so the formatting must happen on the server 
>> side.
>> 
>> However, when the downstream consumer is a library, maybe the library would 
>> like to get the raw millis/nanos since epoch.
>> 
>> Also nested rows and collections might be better encoded with format B for 
>> libraries but interactive sessions are happy if nested types are already 
>> formatted server-side, so not every client needs custom code for the 
>> formatting.
>> 
>> Regards,
>> Timo
>> 
>> 
>> 
>> On 06.12.22 15:13, godfrey he wrote:
>>> Hi, zeklin
>>>> The CLI will use default print style for the non-query result.
>>> Please make sure the print results of EXPLAIN/DESC/SHOW CREATE TABLE
>>> commands are clear.
>>>> We think it’s better to add the root cause to the ErrorResponseBody.
>>> LGTM
>>> Best,
>>> Godfrey
>>> yu zelin  于2022年12月6日周二 17:51写道：
>>>> 
>>>> Hi, Godfrey
>>>> 
>>>> Thanks for your feedback. Below is my thoughts about your questions.
>>>> 
>>>> 1. About RowFormat.
>>>> I agree to your opinion. So we decided to revert the RowFormat related 
>>>> changes
>>>> and let the client to resolve the print format.
>>>> 
>>>> 2. About ContentType
>>>> I agree that the definition of the ContentType is not clear. But how to 
>>>> define the
>>>> statement type is another big question. So, we decided to only tell the 
>>>> query result
>>>> and non-query result apart. The CLI will use default print style for the 
>>>> non-query
>>>> result.
>>>> 
>>>> 3. About ErrorHandling
>>>> I think reuse the current ErrorResponseBody is good, but parse the root 
>>>> cause
>>>> from the exception stack strings is quite hacking. We think it’s better to 
>>>> add the
>>>> root cause to the ErrorResponseBody.
>>>> 
>>>> 4. About Runtime REST API Modifications
>>>> I agree, too. This part is moved to the ‘Future Work’.
>>>> 
>>>> Best,
>>>> Yu Zelin
>>>> 
>>>> 
>>>>> 2022年12月5日 18:33，godfrey he  写道：
>>>>> 
>>>>> Hi

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, Timo,

Thanks for your suggestion. Recently I have discussed with @Godfrey He, 
@Shengkai Fang 
and @Jark Wu about the `RowFormat` (Thanks for all your suggestions). We 
finally came to 
a consensus which is similar to your suggestion. The details are as follows:

1. Add a REST query parameter ‘RowFormat’ = JSON/PLAIN_TEXT to tell the REST 
Endpoint
how to deserialize the RowData int ResultSet.

JSON format means the RowData will be serialized to JSON format, which 
contains original 
LogicalType information, so it can be deserialized back to RowData.

PLAIN_TEXT format means the RowData will be serialized to SQL-compliant, 
plain strings. 
The SQL Client can print the strings directly.

The example URI for fetching results is:
> /v2/sessions/:session_handle/operations/:operation_handle/result/:token?rowFormat=PLAIN_TEXT

2. Introduce two response bodies for fetching results in two formats.

For more details, please take a look at the FLIP 
[https://cwiki.apache.org/confluence/x/T48ODg]. 
I have updated it with an example of query response bodies in two format in 
section:
Public Interface -> REST Endpoint Modification.

> 2022年12月12日 18:09，Timo Walther  写道：
> 
> Hi everyone,
> 
> sorry to jump into this discussion so late.
> 
> > So we decided to revert the RowFormat related changes and let the client to 
> > resolve the print format.
> 
> Could you elaborate a bit on this topic in the FLIP? I still believe that we 
> need 2 types of output formats.
> 
> Format A: for the SQL Client CLI and other interactive notebooks that just 
> uses SQL CAST(... AS STRING) semantics executed on the server side
> 
> Format B: for JDBC SDK or other machine-readable downstream libraries
> 
> Take a TIMESTAMP WITH LOCAL TIME ZONE as an example. The string 
> representation depends on a session configuration option. Clients might not 
> be aware of this session option, so the formatting must happen on the server 
> side.
> 
> However, when the downstream consumer is a library, maybe the library would 
> like to get the raw millis/nanos since epoch.
> 
> Also nested rows and collections might be better encoded with format B for 
> libraries but interactive sessions are happy if nested types are already 
> formatted server-side, so not every client needs custom code for the 
> formatting.
> 
> Regards,
> Timo
> 
> 
> 
> On 06.12.22 15:13, godfrey he wrote:
>> Hi, zeklin
>>> The CLI will use default print style for the non-query result.
>> Please make sure the print results of EXPLAIN/DESC/SHOW CREATE TABLE
>> commands are clear.
>>> We think it’s better to add the root cause to the ErrorResponseBody.
>> LGTM
>> Best,
>> Godfrey
>> yu zelin  于2022年12月6日周二 17:51写道：
>>> 
>>> Hi, Godfrey
>>> 
>>> Thanks for your feedback. Below is my thoughts about your questions.
>>> 
>>> 1. About RowFormat.
>>> I agree to your opinion. So we decided to revert the RowFormat related 
>>> changes
>>> and let the client to resolve the print format.
>>> 
>>> 2. About ContentType
>>> I agree that the definition of the ContentType is not clear. But how to 
>>> define the
>>> statement type is another big question. So, we decided to only tell the 
>>> query result
>>> and non-query result apart. The CLI will use default print style for the 
>>> non-query
>>> result.
>>> 
>>> 3. About ErrorHandling
>>> I think reuse the current ErrorResponseBody is good, but parse the root 
>>> cause
>>> from the exception stack strings is quite hacking. We think it’s better to 
>>> add the
>>> root cause to the ErrorResponseBody.
>>> 
>>> 4. About Runtime REST API Modifications
>>> I agree, too. This part is moved to the ‘Future Work’.
>>> 
>>> Best,
>>> Yu Zelin
>>> 
>>> 
>>>> 2022年12月5日 18:33，godfrey he  写道：
>>>> 
>>>> Hi Zelin,
>>>> 
>>>> Thanks for driving this discussion.
>>>> 
>>>> I have a few comments,
>>>> 
>>>>> Add RowFormat to ResultSet to indicate the format of rows.
>>>> We should not require SqlGateway server to meet the display
>>>> requirements of a CliClient.
>>>> Because different CliClients may have different display style. The
>>>> server just need to response the data,
>>>> and the CliClient prints the result as needed. So RowFormat is not needed.
>>>> 
>>>>> Add ContentType to ResultSet to indicate what kind of data the result 
>>>>> contains.

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, Godfrey

Thanks for your feedback. Below is my thoughts about your questions.

1. About RowFormat.
I agree to your opinion. So we decided to revert the RowFormat related changes 
and let the client to resolve the print format.

2. About ContentType
I agree that the definition of the ContentType is not clear. But how to define 
the 
statement type is another big question. So, we decided to only tell the query 
result 
and non-query result apart. The CLI will use default print style for the 
non-query
result.

3. About ErrorHandling
I think reuse the current ErrorResponseBody is good, but parse the root cause 
from the exception stack strings is quite hacking. We think it’s better to add 
the 
root cause to the ErrorResponseBody.

4. About Runtime REST API Modifications
I agree, too. This part is moved to the ‘Future Work’.

Best,
Yu Zelin


> 2022年12月5日 18:33，godfrey he  写道：
> 
> Hi Zelin,
> 
> Thanks for driving this discussion.
> 
> I have a few comments,
> 
>> Add RowFormat to ResultSet to indicate the format of rows.
> We should not require SqlGateway server to meet the display
> requirements of a CliClient.
> Because different CliClients may have different display style. The
> server just need to response the data,
> and the CliClient prints the result as needed. So RowFormat is not needed.
> 
>> Add ContentType to ResultSet to indicate what kind of data the result 
>> contains.
> from my first sight, the values of ContentType are intersected, such
> as: A select query will return QUERY_RESULT,
> but it also has JOB_ID. OTHER is too ambiguous, I don't know which
> kind of query will return OTHER.
> I recommend returning the concrete type for each statement, such as
> "CREATE TABLE" for "create table xx (...) with ()",
> "SELECT" for "select * from xxx". The statement type can be maintained
> in `Operation`s.
> 
>> Error Handling
> I think current design of error handling mechanism can meet the
> requirement of CliClient, we can get the root cause from
> the stack (see ErrorResponseBody#errors). If it becomes a common
> requirement (for many clients) in the future,
> we can introduce this interface.
> 
>> Runtime REST API Modification for Local Client Migration
> I think this part is over-engineered, this part belongs to optimization.
> The client does not require very high performance, the current design
> can already meet our needs.
> If we find performance problems in the future, do such optimizations.
> 
> Best,
> Godfrey
> 
> yu zelin  于2022年12月5日周一 11:11写道：
>> 
>> Hi, Shammon
>> 
>> Thanks for your feedback. I think it’s good to support jdbc-sdk. However,
>> it's not supported in the gateway side yet. In my opinion, this FLIP is more
>> concerned with the SQL Client. How about put “supporting jdbc-sdk” in
>> ‘Future Work’? We can discuss how to implement it in another thread.
>> 
>> Best,
>> Yu Zelin
>>> 2022年12月2日 18:12，Shammon FY  写道：
>>> 
>>> Hi zelin
>>> 
>>> Thanks for driving this discussion.
>>> 
>>> I notice that the sql-client will interact with sql-gateway by `REST
>>> Client` in the `Executor` in the FLIP, how about introducing jdbc-sdk for
>>> sql-gateway?
>>> 
>>> Then the sql-client can connect the gateway with jdbc-sdk, on the other
>>> hand, the other applications and tools such as jmeter can use the jdbc-sdk
>>> to connect sql-gateway too.
>>> 
>>> Best,
>>> Shammon
>>> 
>>> 
>>> On Fri, Dec 2, 2022 at 4:10 PM yu zelin  wrote:
>>> 
>>>> Hi Jim,
>>>> 
>>>> Thanks for your feedback!
>>>> 
>>>>> Should this configuration be mentioned in the FLIP?
>>>> 
>>>> Sure.
>>>> 
>>>>> some way for the server to be able to limit the number of requests it
>>>> receives.
>>>> I’m sorry that this FLIP is dedicated in implementing the Remote mode, so
>>>> we
>>>> didn't consider much about this. I think the option is enough currently.
>>>> I will add
>>>> the improvement suggestions to the ‘Future Work’.
>>>> 
>>>>> I wonder if two other options are possible
>>>> 
>>>> To forward the raw format to gateway and then to client is possible. The
>>>> raw
>>>> results from sink is in ‘CollectResultIterator#bufferedResult’. First, we
>>>> can find
>>>> a way to get this result without wrapping it. Second, constructing a
>>>> ‘InternalTypeInfo’.
>>>> We can construct

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, Shammon

Thanks for your feedback. I think it’s good to support jdbc-sdk. However, 
it's not supported in the gateway side yet. In my opinion, this FLIP is more
concerned with the SQL Client. How about put “supporting jdbc-sdk” in 
‘Future Work’? We can discuss how to implement it in another thread.

Best,
Yu Zelin
> 2022年12月2日 18:12，Shammon FY  写道：
> 
> Hi zelin
> 
> Thanks for driving this discussion.
> 
> I notice that the sql-client will interact with sql-gateway by `REST
> Client` in the `Executor` in the FLIP, how about introducing jdbc-sdk for
> sql-gateway?
> 
> Then the sql-client can connect the gateway with jdbc-sdk, on the other
> hand, the other applications and tools such as jmeter can use the jdbc-sdk
> to connect sql-gateway too.
> 
> Best,
> Shammon
> 
> 
> On Fri, Dec 2, 2022 at 4:10 PM yu zelin  wrote:
> 
>> Hi Jim,
>> 
>> Thanks for your feedback!
>> 
>>> Should this configuration be mentioned in the FLIP?
>> 
>> Sure.
>> 
>>> some way for the server to be able to limit the number of requests it
>> receives.
>> I’m sorry that this FLIP is dedicated in implementing the Remote mode, so
>> we
>> didn't consider much about this. I think the option is enough currently.
>> I will add
>> the improvement suggestions to the ‘Future Work’.
>> 
>>> I wonder if two other options are possible
>> 
>> To forward the raw format to gateway and then to client is possible. The
>> raw
>> results from sink is in ‘CollectResultIterator#bufferedResult’. First, we
>> can find
>> a way to get this result without wrapping it. Second, constructing a
>> ‘InternalTypeInfo’.
>> We can construct it using the schema information (data’s logical type).
>> After
>> construction, we can get the ’TypeSerializer’ to deserialize the raw
>> result.
>> 
>> 
>> 
>> 
>>> 2022年12月1日 04:54，Jim Hughes  写道：
>>> 
>>> Hi Yu,
>>> 
>>> Thanks for moving my comments to this thread!  Also, thank you for
>>> answering my questions; it is helping me understand the SQL Gateway
>>> better.
>>> 
>>> 5.
>>>> Our idea is to introduce a new session option (like
>>> 'sql-client.result.fetch-interval') to control
>>> the fetching requests sending frequency. What do you think?
>>> 
>>> Should this configuration be mentioned in the FLIP?
>>> 
>>> One slight concern I have with having 'sql-client.result.fetch-interval'
>> as
>>> a session configuration is that users could set it low and cause the
>> client
>>> to send a large volume of requests to the SQL gateway.
>>> 
>>> Generally, I'd like to see some way for the server to be able to limit
>> the
>>> number of requests it receives.  If that really needs to be done by a
>> proxy
>>> in front of the SQL gateway, that is fine as well.  (To be clear, I don't
>>> think my concern here should be blocking in any way.)
>>> 
>>> 7.
>>>> What is the serialization lifecycle for results?
>>> 
>>> I wonder if two other options are possible:
>>> 3) Could the Gateway just forward the result byte array?  (Or does the
>>> Gateway need to deserialize the response in order to understand it for
>> some
>>> reason?)
>>> 4) Could the JobManager prepare the results in JSON?  (Or similarly could
>>> the Client read the format which the JobManager sends?)
>>> 
>>> Thanks again!
>>> 
>>> Cheers,
>>> 
>>> Jim
>>> 
>>> On Wed, Nov 30, 2022 at 9:40 AM yu zelin  wrote:
>>> 
>>>> Hi, all
>>>> 
>>>> Thanks Jim’s questions below. Here I’d like to reply to them.
>>>> 
>>>>> 1. For the Client Parser, is it going to work with the extended syntax
>>>>> from the Flink Table Store?
>>>>> 
>>>>> 2. Relatedly, what will happen if an older Client tries to handle
>>>> syntax
>>>>> that a newer service supports?  (Suppose I use a 1.17 client with a
>>>> 1.18
>>>>> Gateway/system which has a new keyword.  Is there anything we should
>> be
>>>>> designing for upfront?)
>>>>> 
>>>>> 3. How will client and server version mismatches be handled?  Will a
>>>>> single gateway be able to support multiple endpoint versions?
>>>>> 4. How are commands which change a session handled?  Are those sent
>> 

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi Jim,

Thanks for your feedback!

> Should this configuration be mentioned in the FLIP?

Sure. 

> some way for the server to be able to limit the number of requests it 
> receives.
I’m sorry that this FLIP is dedicated in implementing the Remote mode, so we
didn't consider much about this. I think the option is enough currently.  I 
will add 
the improvement suggestions to the ‘Future Work’.

> I wonder if two other options are possible

To forward the raw format to gateway and then to client is possible. The raw 
results from sink is in ‘CollectResultIterator#bufferedResult’. First, we can 
find 
a way to get this result without wrapping it. Second, constructing a 
‘InternalTypeInfo’.
We can construct it using the schema information (data’s logical type). After 
construction, we can get the ’TypeSerializer’ to deserialize the raw result.




> 2022年12月1日 04:54，Jim Hughes  写道：
> 
> Hi Yu,
> 
> Thanks for moving my comments to this thread!  Also, thank you for
> answering my questions; it is helping me understand the SQL Gateway
> better.
> 
> 5.
>> Our idea is to introduce a new session option (like
> 'sql-client.result.fetch-interval') to control
> the fetching requests sending frequency. What do you think?
> 
> Should this configuration be mentioned in the FLIP?
> 
> One slight concern I have with having 'sql-client.result.fetch-interval' as
> a session configuration is that users could set it low and cause the client
> to send a large volume of requests to the SQL gateway.
> 
> Generally, I'd like to see some way for the server to be able to limit the
> number of requests it receives.  If that really needs to be done by a proxy
> in front of the SQL gateway, that is fine as well.  (To be clear, I don't
> think my concern here should be blocking in any way.)
> 
> 7.
>> What is the serialization lifecycle for results?
> 
> I wonder if two other options are possible:
> 3) Could the Gateway just forward the result byte array?  (Or does the
> Gateway need to deserialize the response in order to understand it for some
> reason?)
> 4) Could the JobManager prepare the results in JSON?  (Or similarly could
> the Client read the format which the JobManager sends?)
> 
> Thanks again!
> 
> Cheers,
> 
> Jim
> 
> On Wed, Nov 30, 2022 at 9:40 AM yu zelin  wrote:
> 
>> Hi, all
>> 
>> Thanks Jim’s questions below. Here I’d like to reply to them.
>> 
>>>  1. For the Client Parser, is it going to work with the extended syntax
>>>  from the Flink Table Store?
>>> 
>>>  2. Relatedly, what will happen if an older Client tries to handle
>> syntax
>>>  that a newer service supports?  (Suppose I use a 1.17 client with a
>> 1.18
>>>  Gateway/system which has a new keyword.  Is there anything we should be
>>>  designing for upfront?)
>>> 
>>>  3. How will client and server version mismatches be handled?  Will a
>>>  single gateway be able to support multiple endpoint versions?
>>>  4. How are commands which change a session handled?  Are those sent via
>>>  an ExecuteStatementRequest?
>>> 
>>>  5. The remote POC uses polling for getting back status and getting back
>>>  results.  Would it be possible to switch to web sockets or some other
>>>  mechanism to avoid polling?  If polling is used for both, the polling
>>>  frequency should be different between local and remote configurations.
>>> 
>>>  6. What does this sentence mean?  "The reason why we didn't get the sql
>>>  type in client side is because it's hard for the lightweight
>> client-level
>>>  parser to recognize some sql type  sql, such as query with CTE.  "
>>> 
>>>  7. What is the serialization lifecycle for results?  It makes sense to
>>>  have some control over whether the gateway returns results as SQL or
>> JSON.
>>>  I'd love to see a way to avoid needing to serialize and deserialize
>> results
>>>  on the SQL Gateway if possible.  I'm still new enough to the project
>> that
>>>  I'm not sure if that's readily possible.  Maybe the SQL Gateway's
>> return
>>>  type can be sent as part of the request so that the JobManager can send
>>>  back results in an advantageous format?
>>> 
>>>  8. Does ErrorType need to be marked as @PublicEvolving?
>>> 
>>> I'm excited for the SQL client to support gateway mode!  Given the change
>>> in design, do you think it'll still be part of the Flink 1.17 release?
>> 
>> 1.  ClientParser can work with new (and unknow

Re: SQL Gateway and SQL Client

[yu zelin]

Hi, Jim,

Thanks for your feedback, your suggestions are very good. I have replied in the 
discussion
mail, please take a look.

> 2022年11月29日 03:18，Jim Hughes  写道：
> 
> Hi Shengkai, Yu,
> 
> Thanks for the FLIP!  I have had a chance to read it, and it looks good.  I
> do have some questions:
> 
> I do like the idea of unifying the approaches so that the code doesn't get
> out of step.
> 
>   1. For the Client Parser, is it going to work with the extended syntax
>   from the Flink Table Store?
> 
>   2. Relatedly, what will happen if an older Client tries to handle syntax
>   that a newer service supports?  (Suppose I use a 1.17 client with a 1.18
>   Gateway/system which has a new keyword.  Is there anything we should be
>   designing for upfront?)
> 
>   3. How will client and server version mismatches be handled?  Will a
>   single gateway be able to support multiple endpoint versions?
>   4. How are commands which change a session handled?  Are those sent via
>   an ExecuteStatementRequest?
> 
>   5. The remote POC uses polling for getting back status and getting back
>   results.  Would it be possible to switch to web sockets or some other
>   mechanism to avoid polling?  If polling is used for both, the polling
>   frequency should be different between local and remote configurations.
> 
>   6. What does this sentence mean?  "The reason why we didn't get the sql
>   type in client side is because it's hard for the lightweight client-level
>   parser to recognize some sql type  sql, such as query with CTE.  "
> 
>   7. What is the serialization lifecycle for results?  It makes sense to
>   have some control over whether the gateway returns results as SQL or JSON.
>   I'd love to see a way to avoid needing to serialize and deserialize results
>   on the SQL Gateway if possible.  I'm still new enough to the project that
>   I'm not sure if that's readily possible.  Maybe the SQL Gateway's return
>   type can be sent as part of the request so that the JobManager can send
>   back results in an advantageous format?
> 
>   8. Does ErrorType need to be marked as @PublicEvolving?
> 
> I'm excited for the SQL client to support gateway mode!  Given the change
> in design, do you think it'll still be part of the Flink 1.17 release?
> 
> Cheers,
> 
> Jim
> 
> On Sun, Nov 27, 2022 at 8:54 PM Shengkai Fang  wrote:
> 
>> Hi, Jim and Alexey.
>> 
>> We have written the proposal[1]. It would be appreciated if you can give us
>> some feedback.
>> 
>> Best,
>> Shengkai
>> 
>> [1]
>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway
>> 
>> yu zelin  于2022年11月24日周四 12:06写道：
>> 
>>> Hi Jim,
>>> Sorry for incorrect message in last reply.
>>>> Shengkai will help to your PR
>>> I mean Shengkai will help to review the PRs. And I will add some of your
>>> suggestion to my design.
>>> 
>>> I think the POC code will be cherry-picked to my new design of SQL
>> Client.
>>>> 2022年11月23日 12:18，yu zelin  写道：
>>>> 
>>>> Hi Jim,
>>>> Sorry for late response. Just another busy week :)
>>>> Last week, I’ve discussed within my team about my design. My teammates
>>> think it’s better to unify the local and remote mode, so I’ve
>> investigated
>>> and redesigned a new plan. I’ll inform you after the rewriting of FLIP
>>> finished (will be soon) and Shengkai will help to your PR.
>>>> 
>>>> Best,
>>>> 
>>>> Yu Zelin
>>>> 
>>>>> 2022年11月22日 02:59，Jim Hughes  写道：
>>>>> 
>>>>> Hi Yu, Shengkai,
>>>>> 
>>>>> As a quick update, I've had a chance to try out Yu's POC and it is
>>> working
>>>>> for me.  (Admittedly, I haven't tried it too extensively; I only tried
>>>>> basic operations.)
>>>>> 
>>>>> From my experiments, I did leave a few comments on
>>>>> https://github.com/apache/flink/pull/20958.
>>>>> 
>>>>> Overall, the PRs I see look pretty good.  Are they going to be merged
>>>>> soon?  Anything else I can do to help?
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Jim
>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

ol 
the fetching requests sending frequency. What do you think?

For more information: we are inclined to keep the polling behavior in this 
version. For streaming 
query, fetching results synchronously may occupy resources of the gateway in a 
long period. 
For example, if the job doesn’t return results for a long time because the 
window has not been 
triggered, the synchronously fetching will keep occupying the connection. In 
asynchronous 
situation, the gateway can return a NOT_READY_RESULT quickly and release the 
resources 
for other clients to use. I think we can make some improvements for the whole 
flow path in the 
future.

6. Sorry for that there is mistakes in this sentence. Let me make it clear.

We proposed to add 'ContentType' to indicates the result is for what kind of 
sql. In this sentence, 
I want to explain why we add 'ContentType' since the ClientParser can recognize 
the sql type too. 
It is because the proposed ClientParser can't recognize complex syntax. For 
example, it can’t 
recognize query with CTE. So the result should carry content type information 
to help the client to 
know the sql type. For example, the 'ContentType.QUERY_RESULT' indicates the 
result is for a 
query statement.

7. 
> What is the serialization lifecycle for results?

1) Sink to JobManager: RowData -> Byte[ ] (serialize)
2) JobManager to Gateway : Byte[ ] -> RowData (deserialize)
3) Gateway sending: RowData -> Byte[ ] (serialized to JSON format)
4) Client receiving   : Byte[ ] -> RowData (deserialize)

>  Maybe the SQL Gateway's return type can be sent as part of the request so 
> that the 
JobManager can send  back results in an advantageous format?

Yes. I think it's an improvement for the Client and Gateway. We have some 
ideas. For example,

1) We can move the Gateway into the JobManager and reduce the Ser/De costs from 
JM to Gateway.
2) Or the Gateway can collect the data from the sink function directly instead 
of JobManager.

But I think we can leave this as a future work and discuss in another thread.

8. Yes.

> Do you think it'll still be part of the Flink 1.17 release?
Yes. We will try our best to finish the work.

Feel free to talk to me if I’m wrong or you have any other questions.


> 2022年11月25日 11:48，yu zelin  写道：
> 
> Hi, all
> 
> I want to initiate a discussion on the FLIP-275: Support Remote SQL Client 
> Based on SQL Gateway[1]. 
> The motivation of this FLIP is that the current SQL Client allows only local 
> connection which can not satisfy 
> the common need of connecting to a remote cluster.
> 
> Since the FLIP-91[2] has introduced SQL Gateway, we proposed to implement the 
> Remote SQL Client 
> based on SQL Gateway. In our design, we proposed two main changes:
> 
> 1. New remote mode client which performs connection to the remote gateway 
> through REST API.
> 2. Migration of the current local mode client. We proposed to refactor the 
> local client based on SQL Gateway 
>to unify the interface for two modes.
> 
> Looking forward to your suggestions.
> 
> Best,
> Yu Zelin
> 
> [1] https://cwiki.apache.org/confluence/x/T48ODg
> [2] https://cwiki.apache.org/confluence/x/rIyMC

[DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

[yu zelin]

Hi, all

I want to initiate a discussion on the FLIP-275: Support Remote SQL Client 
Based on SQL Gateway[1]. 
The motivation of this FLIP is that the current SQL Client allows only local 
connection which can not satisfy 
the common need of connecting to a remote cluster.

Since the FLIP-91[2] has introduced SQL Gateway, we proposed to implement the 
Remote SQL Client 
based on SQL Gateway. In our design, we proposed two main changes:

1. New remote mode client which performs connection to the remote gateway 
through REST API.
2. Migration of the current local mode client. We proposed to refactor the 
local client based on SQL Gateway 
to unify the interface for two modes.

Looking forward to your suggestions.

Best,
Yu Zelin

[1] https://cwiki.apache.org/confluence/x/T48ODg
[2] https://cwiki.apache.org/confluence/x/rIyMC

Re: SQL Gateway and SQL Client

[yu zelin]

Hi Jim,
Sorry for incorrect message in last reply. 
> Shengkai will help to your PR
I mean Shengkai will help to review the PRs. And I will add some of your 
suggestion to my design.

I think the POC code will be cherry-picked to my new design of SQL Client.
> 2022年11月23日 12:18，yu zelin  写道：
> 
> Hi Jim,
> Sorry for late response. Just another busy week :)
> Last week, I’ve discussed within my team about my design. My teammates think 
> it’s better to unify the local and remote mode, so I’ve investigated and 
> redesigned a new plan. I’ll inform you after the rewriting of FLIP finished 
> (will be soon) and Shengkai will help to your PR.
> 
> Best,
> 
> Yu Zelin
> 
>> 2022年11月22日 02:59，Jim Hughes  写道：
>> 
>> Hi Yu, Shengkai,
>> 
>> As a quick update, I've had a chance to try out Yu's POC and it is working
>> for me.  (Admittedly, I haven't tried it too extensively; I only tried
>> basic operations.)
>> 
>> From my experiments, I did leave a few comments on
>> https://github.com/apache/flink/pull/20958.
>> 
>> Overall, the PRs I see look pretty good.  Are they going to be merged
>> soon?  Anything else I can do to help?
>> 
>> Cheers,
>> 
>> Jim
> 

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

[yu zelin]

Congratulations，Godfrey and Xingbo!

Best,
Yu Zelin
> 2022年11月23日 12:18，Dian Fu  写道：
> 
> Hi everyone,
> 
> On behalf of the Apache Flink PMC, I'm very happy to announce that Godfrey
> He and Xingbo Huang have joined the Flink PMC!
> 
> Godfrey He becomes a Flink committer in Sep 2020. His contributions are
> mainly focused on the Flink table module, covering almost all important
> parts such as Client(SQL Client, SQL gateway, JDBC driver, etc), API, SQL
> parser, query optimization, query execution, etc. Especially in the query
> optimization part, he built the query optimization framework and the cost
> model, improved the statistics information and made a lot of optimizations,
> e.g. dynamic partition pruning, join hint, multiple input rewrite, etc. In
> addition, he has done a lot of groundwork, such as refactoring the
> ExecNode(which is the basis for further DAG optimizations) and SQL Plan
> JSON serialization (which is a big step to support SQL job version
> upgrade). Besides that, he's also helping the projects in other ways, e.g.
> managing releases, giving talks, etc.
> 
> Xingbo Huang becomes a Flink committer in Feb 2021. His contributions are
> mainly focused on the PyFlink module and he's the author of many important
> features in PyFlink, e.g. Cython support, Python thread execution mode,
> Python UDTF support, Python UDAF support in windowing, etc. He is also one
> of the main contributors in the Flink ML project. Besides that, he's also
> helping to manage releases, taking care of the build stabilites, etc.
> 
> Congratulations and welcome them as Apache Flink PMC!
> 
> Regards,
> Dian

Re: SQL Gateway and SQL Client

[yu zelin]

Hi Jim,
Sorry for late response. Just another busy week :)
Last week, I’ve discussed within my team about my design. My teammates think 
it’s better to unify the local and remote mode, so I’ve investigated and 
redesigned a new plan. I’ll inform you after the rewriting of FLIP finished 
(will be soon) and Shengkai will help to your PR.

Best,

Yu Zelin

> 2022年11月22日 02:59，Jim Hughes  写道：
> 
> Hi Yu, Shengkai,
> 
> As a quick update, I've had a chance to try out Yu's POC and it is working
> for me.  (Admittedly, I haven't tried it too extensively; I only tried
> basic operations.)
> 
> From my experiments, I did leave a few comments on
> https://github.com/apache/flink/pull/20958.
> 
> Overall, the PRs I see look pretty good.  Are they going to be merged
> soon?  Anything else I can do to help?
> 
> Cheers,
> 
> Jim

Re: SQL Gateway and SQL Client

[yu zelin]

Hi Jim,

It would be nice if you can take a look on 
https://github.com/apache/flink/pull/21133 and give me some feedback.

Best,
Yu Zelin

> 2022年11月12日 00:44，Jim Hughes  写道：
> 
> Hi Shengkai,
> 
> I think there is an additional case where a proxy is between the client and
> gateway.  In that case, being able to pass headers would allow for
> additional options / features.
> 
> I see several PRs from Yu Zelin.  Is there a first one to review?
> 
> Cheers,
> 
> Jim
> 
> On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang  wrote:
> 
>> Hi, Jim.
>> 
>>> how to pass additional headers when sending REST requests
>> 
>> Could you share what headers do you want to send when using SQL Client?  I
>> think there are two cases we need to consider. Please correct me if I am
>> wrong.
>> 
>> # Case 1
>> 
>> If users wants to connect to the SQL Gateway with its password, I think the
>> users should type
>> ```
>> ./sql-client.sh --user xxx --password xxx
>> ```
>> in the terminal and the OpenSessionRequest should be enough.
>> 
>> # Case 2
>> 
>> If users  wants to modify the execution config, users should type
>> ```
>> Flink SQL> SET  `` = ``;
>> ```
>> in the terminal. The Client can send ExecuteStatementRequest to the
>> Gateway.
>> 
>>> As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know.
>> 
>> It would be nice you can join us to finish the feature. I think the
>> modification about the SQL Gateway side is ready to review.
>> 
>> Best,
>> Shengkai
>> 
>> 
>> Jim Hughes  于2022年11月11日周五 05:19写道：
>> 
>>> Hi Yu Zelin,
>>> 
>>> I have read through your draft and it looks good.  I am new to Flink, so
>> I
>>> haven't learned about everything which needs to be done yet.
>>> 
>>> One of the considerations that I'm interested in understanding is how to
>>> pass additional headers when sending REST requests.  From looking at the
>>> code, it looks like a custom `OutboundChannelHandlerFactory` could be
>>> created to read additional configuration and set headers.  Does that make
>>> sense?
>>> 
>>> Thank you very much for sharing the proof of concept code and the
>>> document.  As you have FLIPs or PRs, feel free to let me, Jamie, and
>> Alexey
>>> know.  We'll be happy to review them.
>>> 
>>> Cheers,
>>> 
>>> Jim
>>> 
>>> On Wed, Nov 9, 2022 at 11:43 PM yu zelin  wrote:
>>> 
>>>> Hi, all
>>>> Sorry for late response. As Shengkai mentioned, Currently I’m working
>>> with
>>>> him on SQL Client, dedicating to implement the Remote Mode of SQL
>>> Client. I
>>>> have written a draft of implementation plan and will write a FLIP about
>>> it
>>>> ASAP. If you are interested in, please take a look at the draft and
>> it’s
>>>> nice if you give me some feedback.
>>>> The doc is at:
>>>> 
>>> 
>> https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing
>>>> 
>>>>> 2022年11月7日 11:19，Shengkai Fang  写道：
>>>>> 
>>>>> Hi, all. Sorry for the late reply.
>>>>> 
>>>>>> Is the gateway mode planned to be supported for SQL Client in 1.17?
>>>>>> Do you have anything you can already share so we can start with
>> your
>>>> work or just play around with it.
>>>>> 
>>>>> Yes. @yzl is working on it and he will list the implementation plan
>>>> later and share the progress. I think the change is not very large and
>> I
>>>> think it's not a big problem to finish this in the release-1.17. I will
>>>> join to develop this in the mid of November.
>>>>> 
>>>>> Best,
>>>>> Shengkai
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Jamie Grier mailto:jgr...@apache.org>>
>>>> 于2022年11月5日周六 00:48写道：
>>>>>> Hi Shengkai,
>>>>>> 
>>>>>> We're doing more and more Flink development at Confluent these days
>>> and
>>>> we're currently trying to bootstrap a prototype that relies on the SQL
>>>> Client and Gateway.  We will be using the the components in some of our
>>>> projects and would like to co-develop them with you and the rest of the
>>>>

Re: SQL Gateway and SQL Client

[yu zelin]

Hi, all
Sorry for late response. As Shengkai mentioned, Currently I’m working with him 
on SQL Client, dedicating to implement the Remote Mode of SQL Client. I have 
written a draft of implementation plan and will write a FLIP about it ASAP. If 
you are interested in, please take a look at the draft and it’s nice if you 
give me some feedback. 
The doc is at: 
https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing

> 2022年11月7日 11:19，Shengkai Fang  写道：
> 
> Hi, all. Sorry for the late reply. 
> 
> > Is the gateway mode planned to be supported for SQL Client in 1.17?
> > Do you have anything you can already share so we can start with your work 
> > or just play around with it.  
> 
> Yes. @yzl is working on it and he will list the implementation plan later and 
> share the progress. I think the change is not very large and I think it's not 
> a big problem to finish this in the release-1.17. I will join to develop this 
> in the mid of November.
> 
> Best,
> Shengkai
> 
> 
> 
> 
> Jamie Grier mailto:jgr...@apache.org>> 于2022年11月5日周六 
> 00:48写道：
>> Hi Shengkai,
>> 
>> We're doing more and more Flink development at Confluent these days and 
>> we're currently trying to bootstrap a prototype that relies on the SQL 
>> Client and Gateway.  We will be using the the components in some of our 
>> projects and would like to co-develop them with you and the rest of the 
>> Flink community.
>> 
>> As of right now it's a pretty big blocker for our upcoming milestone that 
>> the SQL Client has not yet been modified to talk to the SQL Gateway and we 
>> want to help with this effort ASAP!  We would be even willing to take over 
>> the work if it's not yet started but I suspect it already is.
>> 
>> Anyway, rather than start working immediately on the SQL Client and adding a 
>> the new Gateway mode ourselves we wanted to start a conversation with you 
>> and see where you're at with things and offer to help.
>> 
>> Do you have anything you can already share so we can start with your work or 
>> just play around with it.  Like I said, we just want to get started and are 
>> very able to help in this area.  We see both the SQL Client and Gateway 
>> being very important for us and have a good team to help develop it.
>> 
>> Let me know if there is a branch you can share, etc.  It would be much 
>> appreciated!
>> 
>> -Jamie Grier
>> 
>> 
>> On 2022/10/28 06:06:49 Shengkai Fang wrote:
>> > Hi.
>> > 
>> > > Is there a possibility for us to get engaged and at least introduce
>> > initial changes to support authentication/authorization?
>> > 
>> > Yes. You can write a FLIP about the design and change. We can discuss this
>> > in the dev mail. If the FLIP passes, we can develop it together.
>> > 
>> > > Another question about persistent Gateway: did you have any specific
>> > thoughts about it or some draft design?
>> > 
>> > We don't have any detailed plan about this. But I know Livy has a similar
>> > feature.
>> > 
>> > Best,
>> > Shengkai
>> > 
>> > 
>> > Alexey Leonov-Vendrovskiy > > > 于2022年10月27日周四 15:12写道：
>> > 
>> > > Apologies from the delayed response on my side.
>> > >
>> > >  I think the authentication module is not part of our plan in 1.17 
>> > > because
>> > >> of the busy work. I think we'll start the design at the end of the
>> > >> release-1.17.
>> > >
>> > >
>> > > Is there a possibility for us to get engaged and at least introduce
>> > > initial changes to support authentication/authorization? Specifically,
>> > > changes in the API and in SQL Client.
>> > >
>> > > We expect the following authentication flow:
>> > >
>> > > On the SQL gateway we want to be able to use a delegation token.
>> > > SQL client should be able to supply an API key.
>> > > The SQL Gateway *would not *be submitting jobs on behalf of the client.
>> > >
>> > > Ideally it would be nice to introduce some interfaces in the SQL Gateway
>> > > that would allow implementing custom authentication and authorization.
>> > >
>> > > Another question about persistent Gateway: did you have any specific
>> > > thoughts about it or some draft design?
>> > >
>> > > Thanks,
>> > > Alexey
>> > >
>> > >
>> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang > > > > wrote:
>> > >
>> > >> Sorry for the late response.
>> > >>
>> > >> In the next version(Flink 1.17), we plan to support the SQL Client to
>> > >> submit the statement to the Flink SQL Gateway. The FLINK-29486
>> > >>  is the first step to
>> > >> remove the usage of the `Parser` in the client side, which needs to read
>> > >> the table schema during the converting sql node to operation. I think 
>> > >> the authentication
>> > >> module is not part of our plan in 1.17 because of the busy work. I think
>> > >> we'll start the design at the end of the release-1.17.
>> > >> But could you share more details about the requirements of the
>> > >> authenti