Hi Shengkai, Thanks! Can you say more about how the RestClient sets the headers? I looked at the code, and I only see headers being set in the RestClient here[1].
I'm hoping to set arbitrary headers from configuration. I'm not seeing a way to do that presently. [1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java#L410-L425 Cheers, Jim On Sun, Nov 13, 2022 at 10:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > Hi Jim, > > Thanks for your input. You can look here[1] on the server side. You can > modify the return type of the handleRequest to Tuple2<Response, Headers> to > get session-specific headers. RestClient has already exposed a method to > send requests with headers for the client side. > > [1] > > https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/rest/handler/AbstractSqlGatewayRestHandler.java#L98 > > Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月12日周六 00:45写道： > > > Hi Shengkai, > > > > I think there is an additional case where a proxy is between the client > and > > gateway. In that case, being able to pass headers would allow for > > additional options / features. > > > > I see several PRs from Yu Zelin. Is there a first one to review? > > > > Cheers, > > > > Jim > > > > On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > > > > > Hi, Jim. > > > > > > > how to pass additional headers when sending REST requests > > > > > > Could you share what headers do you want to send when using SQL Client? > > I > > > think there are two cases we need to consider. Please correct me if I > am > > > wrong. > > > > > > # Case 1 > > > > > > If users wants to connect to the SQL Gateway with its password, I think > > the > > > users should type > > > ``` > > > ./sql-client.sh --user xxx --password xxx > > > ``` > > > in the terminal and the OpenSessionRequest should be enough. > > > > > > # Case 2 > > > > > > If users wants to modify the execution config, users should type > > > ``` > > > Flink SQL> SET `<config_name>` = `<config_value>`; > > > ``` > > > in the terminal. The Client can send ExecuteStatementRequest to the > > > Gateway. > > > > > > > As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey > know. > > > > > > It would be nice you can join us to finish the feature. I think the > > > modification about the SQL Gateway side is ready to review. > > > > > > Best, > > > Shengkai > > > > > > > > > Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： > > > > > > > Hi Yu Zelin, > > > > > > > > I have read through your draft and it looks good. I am new to Flink, > > so > > > I > > > > haven't learned about everything which needs to be done yet. > > > > > > > > One of the considerations that I'm interested in understanding is how > > to > > > > pass additional headers when sending REST requests. From looking at > > the > > > > code, it looks like a custom `OutboundChannelHandlerFactory` could be > > > > created to read additional configuration and set headers. Does that > > make > > > > sense? > > > > > > > > Thank you very much for sharing the proof of concept code and the > > > > document. As you have FLIPs or PRs, feel free to let me, Jamie, and > > > Alexey > > > > know. We'll be happy to review them. > > > > > > > > Cheers, > > > > > > > > Jim > > > > > > > > On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> > > wrote: > > > > > > > > > Hi, all > > > > > Sorry for late response. As Shengkai mentioned, Currently I’m > working > > > > with > > > > > him on SQL Client, dedicating to implement the Remote Mode of SQL > > > > Client. I > > > > > have written a draft of implementation plan and will write a FLIP > > about > > > > it > > > > > ASAP. If you are interested in, please take a look at the draft and > > > it’s > > > > > nice if you give me some feedback. > > > > > The doc is at: > > > > > > > > > > > > > > > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing > > > > > > > > > > > 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > > > > > > > > > > > > Hi, all. Sorry for the late reply. > > > > > > > > > > > > > Is the gateway mode planned to be supported for SQL Client in > > 1.17? > > > > > > > Do you have anything you can already share so we can start with > > > your > > > > > work or just play around with it. > > > > > > > > > > > > Yes. @yzl is working on it and he will list the implementation > plan > > > > > later and share the progress. I think the change is not very large > > and > > > I > > > > > think it's not a big problem to finish this in the release-1.17. I > > will > > > > > join to develop this in the mid of November. > > > > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> > > > > > 于2022年11月5日周六 00:48写道： > > > > > >> Hi Shengkai, > > > > > >> > > > > > >> We're doing more and more Flink development at Confluent these > > days > > > > and > > > > > we're currently trying to bootstrap a prototype that relies on the > > SQL > > > > > Client and Gateway. We will be using the the components in some of > > our > > > > > projects and would like to co-develop them with you and the rest of > > the > > > > > Flink community. > > > > > >> > > > > > >> As of right now it's a pretty big blocker for our upcoming > > milestone > > > > > that the SQL Client has not yet been modified to talk to the SQL > > > Gateway > > > > > and we want to help with this effort ASAP! We would be even > willing > > to > > > > > take over the work if it's not yet started but I suspect it already > > is. > > > > > >> > > > > > >> Anyway, rather than start working immediately on the SQL Client > > and > > > > > adding a the new Gateway mode ourselves we wanted to start a > > > conversation > > > > > with you and see where you're at with things and offer to help. > > > > > >> > > > > > >> Do you have anything you can already share so we can start with > > your > > > > > work or just play around with it. Like I said, we just want to get > > > > started > > > > > and are very able to help in this area. We see both the SQL Client > > and > > > > > Gateway being very important for us and have a good team to help > > > develop > > > > it. > > > > > >> > > > > > >> Let me know if there is a branch you can share, etc. It would > be > > > much > > > > > appreciated! > > > > > >> > > > > > >> -Jamie Grier > > > > > >> > > > > > >> > > > > > >> On 2022/10/28 06:06:49 Shengkai Fang wrote: > > > > > >> > Hi. > > > > > >> > > > > > > >> > > Is there a possibility for us to get engaged and at least > > > > introduce > > > > > >> > initial changes to support authentication/authorization? > > > > > >> > > > > > > >> > Yes. You can write a FLIP about the design and change. We can > > > > discuss > > > > > this > > > > > >> > in the dev mail. If the FLIP passes, we can develop it > together. > > > > > >> > > > > > > >> > > Another question about persistent Gateway: did you have any > > > > specific > > > > > >> > thoughts about it or some draft design? > > > > > >> > > > > > > >> > We don't have any detailed plan about this. But I know Livy > has > > a > > > > > similar > > > > > >> > feature. > > > > > >> > > > > > > >> > Best, > > > > > >> > Shengkai > > > > > >> > > > > > > >> > > > > > > >> > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > > > vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： > > > > > >> > > > > > > >> > > Apologies from the delayed response on my side. > > > > > >> > > > > > > > >> > > I think the authentication module is not part of our plan > in > > > 1.17 > > > > > because > > > > > >> > >> of the busy work. I think we'll start the design at the end > > of > > > > the > > > > > >> > >> release-1.17. > > > > > >> > > > > > > > >> > > > > > > > >> > > Is there a possibility for us to get engaged and at least > > > > introduce > > > > > >> > > initial changes to support authentication/authorization? > > > > > Specifically, > > > > > >> > > changes in the API and in SQL Client. > > > > > >> > > > > > > > >> > > We expect the following authentication flow: > > > > > >> > > > > > > > >> > > On the SQL gateway we want to be able to use a delegation > > token. > > > > > >> > > SQL client should be able to supply an API key. > > > > > >> > > The SQL Gateway *would not *be submitting jobs on behalf of > > the > > > > > client. > > > > > >> > > > > > > > >> > > Ideally it would be nice to introduce some interfaces in the > > SQL > > > > > Gateway > > > > > >> > > that would allow implementing custom authentication and > > > > > authorization. > > > > > >> > > > > > > > >> > > Another question about persistent Gateway: did you have any > > > > specific > > > > > >> > > thoughts about it or some draft design? > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > Alexey > > > > > >> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < > > > fskm...@gmail.com > > > > > <mailto:fskm...@gmail.com>> wrote: > > > > > >> > > > > > > > >> > >> Sorry for the late response. > > > > > >> > >> > > > > > >> > >> In the next version(Flink 1.17), we plan to support the SQL > > > > Client > > > > > to > > > > > >> > >> submit the statement to the Flink SQL Gateway. The > > FLINK-29486 > > > > > >> > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the > > > first > > > > > step to > > > > > >> > >> remove the usage of the `Parser` in the client side, which > > > needs > > > > > to read > > > > > >> > >> the table schema during the converting sql node to > > operation. I > > > > > think the authentication > > > > > >> > >> module is not part of our plan in 1.17 because of the busy > > > work. > > > > I > > > > > think > > > > > >> > >> we'll start the design at the end of the release-1.17. > > > > > >> > >> But could you share more details about the requirements of > > the > > > > > >> > >> authentication? > > > > > >> > >> - Do you use the kerberos or delegation token or password > to > > do > > > > the > > > > > >> > >> authentication? > > > > > >> > >> - After the authentication, do you need the sql gateway to > > > submit > > > > > the > > > > > >> > >> job on behalf of the client? > > > > > >> > >> - ... > > > > > >> > >> > > > > > >> > >> For detailed implementation, I think Hive and Presto are > good > > > > > examples to > > > > > >> > >> dig in. If you have some thoughts about the authentication > > > > module, > > > > > >> > >> please let me know. > > > > > >> > >> > > > > > >> > >> Best, > > > > > >> > >> Shengkai > > > > > >> > >> > > > > > >> > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > > > vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： > > > > > >> > >> > > > > > >> > >>> Thank you for the response, Yuxia! > > > > > >> > >>> > > > > > >> > >>> Shengkai, I would like to learn more about nearest and a > bit > > > > more > > > > > >> > >>> distant plans about development of the SQL Gateway and the > > SQL > > > > > Client. > > > > > >> > >>> Do you have a description of the work planned or maybe can > > > share > > > > > general > > > > > >> > >>> thoughts about the Authentication module, or Persistent > > > Gateway. > > > > > >> > >>> How can the authentication part be addressed on the SQL > > Client > > > > > side? > > > > > >> > >>> > > > > > >> > >>> Regards, > > > > > >> > >>> -Alexey > > > > > >> > >>> > > > > > >> > >>> > > > > > >> > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < > > > > > luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> > > > > > >> > >>> wrote: > > > > > >> > >>> > > > > > >> > >>>> > In what Flink’s release the connection from SQL Client > to > > > the > > > > > Gateway > > > > > >> > >>>> is > > > > > >> > >>>> expected to be added? > > > > > >> > >>>> Flink 1.17 > > > > > >> > >>>> > > > > > >> > >>>> > “Authentication module” (2) and “Persistent Gateway” > (4) > > as > > > > > >> > >>>> possible future work. Were there any recent discussions > on > > > > these > > > > > >> > >>>> subjects? > > > > > >> > >>>> No recent discussions on these subjects, but I think > it'll > > > come > > > > > in > > > > > >> > >>>> Flink 1.17 > > > > > >> > >>>> > > > > > >> > >>>> > Another related topic: are there ideas around making > SQL > > > > > Gateway a > > > > > >> > >>>> multi-tenant > > > > > >> > >>>> component? > > > > > >> > >>>> Yes. > > > > > >> > >>>> > > > > > >> > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, > > > maybe > > > > > he can > > > > > >> > >>>> provide more information. > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> Best regards, > > > > > >> > >>>> Yuxia > > > > > >> > >>>> > > > > > >> > >>>> ----- 原始邮件 ----- > > > > > >> > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com > > > > <mailto: > > > > > vendrov...@gmail.com>> > > > > > >> > >>>> 收件人: "dev" <dev@flink.apache.org <mailto: > > > dev@flink.apache.org > > > > >> > > > > > >> > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > > > > > >> > >>>> 主题: SQL Gateway and SQL Client > > > > > >> > >>>> > > > > > >> > >>>> Hi all, > > > > > >> > >>>> > > > > > >> > >>>> I’m Alexey from Confluent. This is my first email in this > > > > > discussion > > > > > >> > >>>> list. > > > > > >> > >>>> I’m rather new to Flink, and to local customs of > > > communication. > > > > > I want > > > > > >> > >>>> to > > > > > >> > >>>> dive deeper and hopefully get more involved over time. > > > > > >> > >>>> > > > > > >> > >>>> Currently I have a few questions around SQL Gateway and > SQL > > > > > Client. > > > > > >> > >>>> Specifically I wanted to learn what is the vision around > > the > > > > > nearest > > > > > >> > >>>> future > > > > > >> > >>>> of these two components. > > > > > >> > >>>> > > > > > >> > >>>> In what Flink’s release the connection from SQL Client to > > the > > > > > Gateway is > > > > > >> > >>>> expected to be added? I was looking at > > > > > >> > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and > > > > recently > > > > > it got > > > > > >> > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in > > > > Remote > > > > > Mode” > > > > > >> > >>>> to > > > > > >> > >>>> “Introduce Client Parser to get statement type”. I did > > some > > > > > search, but > > > > > >> > >>>> didn’t find a good place where the client's work in this > > > > > direction is > > > > > >> > >>>> discussed or tracked. > > > > > >> > >>>> > > > > > >> > >>>> A couple questions about the SQL Gateway. The FLIP-91 > > > > > >> > >>>> < > > > > > >> > >>>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > > > > > >> > >>>> > > > > > > >> > >>>> mentions “Authentication module” (2) and “Persistent > > Gateway” > > > > > (4) as > > > > > >> > >>>> possible future work. Were there any recent discussions > on > > > > these > > > > > >> > >>>> subjects? > > > > > >> > >>>> Or maybe there are some ideas how to move these > directions > > > > > forward? > > > > > >> > >>>> Another > > > > > >> > >>>> related topic: are there ideas around making SQL Gateway > a > > > > > multi-tenant > > > > > >> > >>>> component? > > > > > >> > >>>> > > > > > >> > >>>> Thank you, > > > > > >> > >>>> > > > > > >> > >>>> Alexey > > > > > >> > >>>> > > > > > >> > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
