Hi Jim, It would be nice if you can take a look on https://github.com/apache/flink/pull/21133 and give me some feedback.
Best, Yu Zelin > 2022年11月12日 00:44，Jim Hughes <jhug...@confluent.io.INVALID> 写道： > > Hi Shengkai, > > I think there is an additional case where a proxy is between the client and > gateway. In that case, being able to pass headers would allow for > additional options / features. > > I see several PRs from Yu Zelin. Is there a first one to review? > > Cheers, > > Jim > > On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > >> Hi, Jim. >> >>> how to pass additional headers when sending REST requests >> >> Could you share what headers do you want to send when using SQL Client? I >> think there are two cases we need to consider. Please correct me if I am >> wrong. >> >> # Case 1 >> >> If users wants to connect to the SQL Gateway with its password, I think the >> users should type >> ``` >> ./sql-client.sh --user xxx --password xxx >> ``` >> in the terminal and the OpenSessionRequest should be enough. >> >> # Case 2 >> >> If users wants to modify the execution config, users should type >> ``` >> Flink SQL> SET `<config_name>` = `<config_value>`; >> ``` >> in the terminal. The Client can send ExecuteStatementRequest to the >> Gateway. >> >>> As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. >> >> It would be nice you can join us to finish the feature. I think the >> modification about the SQL Gateway side is ready to review. >> >> Best, >> Shengkai >> >> >> Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： >> >>> Hi Yu Zelin, >>> >>> I have read through your draft and it looks good. I am new to Flink, so >> I >>> haven't learned about everything which needs to be done yet. >>> >>> One of the considerations that I'm interested in understanding is how to >>> pass additional headers when sending REST requests. From looking at the >>> code, it looks like a custom `OutboundChannelHandlerFactory` could be >>> created to read additional configuration and set headers. Does that make >>> sense? >>> >>> Thank you very much for sharing the proof of concept code and the >>> document. As you have FLIPs or PRs, feel free to let me, Jamie, and >> Alexey >>> know. We'll be happy to review them. >>> >>> Cheers, >>> >>> Jim >>> >>> On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> wrote: >>> >>>> Hi, all >>>> Sorry for late response. As Shengkai mentioned, Currently I’m working >>> with >>>> him on SQL Client, dedicating to implement the Remote Mode of SQL >>> Client. I >>>> have written a draft of implementation plan and will write a FLIP about >>> it >>>> ASAP. If you are interested in, please take a look at the draft and >> it’s >>>> nice if you give me some feedback. >>>> The doc is at: >>>> >>> >> https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing >>>> >>>>> 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： >>>>> >>>>> Hi, all. Sorry for the late reply. >>>>> >>>>>> Is the gateway mode planned to be supported for SQL Client in 1.17? >>>>>> Do you have anything you can already share so we can start with >> your >>>> work or just play around with it. >>>>> >>>>> Yes. @yzl is working on it and he will list the implementation plan >>>> later and share the progress. I think the change is not very large and >> I >>>> think it's not a big problem to finish this in the release-1.17. I will >>>> join to develop this in the mid of November. >>>>> >>>>> Best, >>>>> Shengkai >>>>> >>>>> >>>>> >>>>> >>>>> Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> >>>> 于2022年11月5日周六 00:48写道： >>>>>> Hi Shengkai, >>>>>> >>>>>> We're doing more and more Flink development at Confluent these days >>> and >>>> we're currently trying to bootstrap a prototype that relies on the SQL >>>> Client and Gateway. We will be using the the components in some of our >>>> projects and would like to co-develop them with you and the rest of the >>>> Flink community. >>>>>> >>>>>> As of right now it's a pretty big blocker for our upcoming milestone >>>> that the SQL Client has not yet been modified to talk to the SQL >> Gateway >>>> and we want to help with this effort ASAP! We would be even willing to >>>> take over the work if it's not yet started but I suspect it already is. >>>>>> >>>>>> Anyway, rather than start working immediately on the SQL Client and >>>> adding a the new Gateway mode ourselves we wanted to start a >> conversation >>>> with you and see where you're at with things and offer to help. >>>>>> >>>>>> Do you have anything you can already share so we can start with your >>>> work or just play around with it. Like I said, we just want to get >>> started >>>> and are very able to help in this area. We see both the SQL Client and >>>> Gateway being very important for us and have a good team to help >> develop >>> it. >>>>>> >>>>>> Let me know if there is a branch you can share, etc. It would be >> much >>>> appreciated! >>>>>> >>>>>> -Jamie Grier >>>>>> >>>>>> >>>>>> On 2022/10/28 06:06:49 Shengkai Fang wrote: >>>>>>> Hi. >>>>>>> >>>>>>>> Is there a possibility for us to get engaged and at least >>> introduce >>>>>>> initial changes to support authentication/authorization? >>>>>>> >>>>>>> Yes. You can write a FLIP about the design and change. We can >>> discuss >>>> this >>>>>>> in the dev mail. If the FLIP passes, we can develop it together. >>>>>>> >>>>>>>> Another question about persistent Gateway: did you have any >>> specific >>>>>>> thoughts about it or some draft design? >>>>>>> >>>>>>> We don't have any detailed plan about this. But I know Livy has a >>>> similar >>>>>>> feature. >>>>>>> >>>>>>> Best, >>>>>>> Shengkai >>>>>>> >>>>>>> >>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: >>>> vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： >>>>>>> >>>>>>>> Apologies from the delayed response on my side. >>>>>>>> >>>>>>>> I think the authentication module is not part of our plan in >> 1.17 >>>> because >>>>>>>>> of the busy work. I think we'll start the design at the end of >>> the >>>>>>>>> release-1.17. >>>>>>>> >>>>>>>> >>>>>>>> Is there a possibility for us to get engaged and at least >>> introduce >>>>>>>> initial changes to support authentication/authorization? >>>> Specifically, >>>>>>>> changes in the API and in SQL Client. >>>>>>>> >>>>>>>> We expect the following authentication flow: >>>>>>>> >>>>>>>> On the SQL gateway we want to be able to use a delegation token. >>>>>>>> SQL client should be able to supply an API key. >>>>>>>> The SQL Gateway *would not *be submitting jobs on behalf of the >>>> client. >>>>>>>> >>>>>>>> Ideally it would be nice to introduce some interfaces in the SQL >>>> Gateway >>>>>>>> that would allow implementing custom authentication and >>>> authorization. >>>>>>>> >>>>>>>> Another question about persistent Gateway: did you have any >>> specific >>>>>>>> thoughts about it or some draft design? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Alexey >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < >> fskm...@gmail.com >>>> <mailto:fskm...@gmail.com>> wrote: >>>>>>>> >>>>>>>>> Sorry for the late response. >>>>>>>>> >>>>>>>>> In the next version(Flink 1.17), we plan to support the SQL >>> Client >>>> to >>>>>>>>> submit the statement to the Flink SQL Gateway. The FLINK-29486 >>>>>>>>> <https://issues.apache.org/jira/browse/FLINK-29486> is the >> first >>>> step to >>>>>>>>> remove the usage of the `Parser` in the client side, which >> needs >>>> to read >>>>>>>>> the table schema during the converting sql node to operation. I >>>> think the authentication >>>>>>>>> module is not part of our plan in 1.17 because of the busy >> work. >>> I >>>> think >>>>>>>>> we'll start the design at the end of the release-1.17. >>>>>>>>> But could you share more details about the requirements of the >>>>>>>>> authentication? >>>>>>>>> - Do you use the kerberos or delegation token or password to do >>> the >>>>>>>>> authentication? >>>>>>>>> - After the authentication, do you need the sql gateway to >> submit >>>> the >>>>>>>>> job on behalf of the client? >>>>>>>>> - ... >>>>>>>>> >>>>>>>>> For detailed implementation, I think Hive and Presto are good >>>> examples to >>>>>>>>> dig in. If you have some thoughts about the authentication >>> module, >>>>>>>>> please let me know. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Shengkai >>>>>>>>> >>>>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: >>>> vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： >>>>>>>>> >>>>>>>>>> Thank you for the response, Yuxia! >>>>>>>>>> >>>>>>>>>> Shengkai, I would like to learn more about nearest and a bit >>> more >>>>>>>>>> distant plans about development of the SQL Gateway and the SQL >>>> Client. >>>>>>>>>> Do you have a description of the work planned or maybe can >> share >>>> general >>>>>>>>>> thoughts about the Authentication module, or Persistent >> Gateway. >>>>>>>>>> How can the authentication part be addressed on the SQL Client >>>> side? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> -Alexey >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < >>>> luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>>> In what Flink’s release the connection from SQL Client to >> the >>>> Gateway >>>>>>>>>>> is >>>>>>>>>>> expected to be added? >>>>>>>>>>> Flink 1.17 >>>>>>>>>>> >>>>>>>>>>>> “Authentication module” (2) and “Persistent Gateway” (4) as >>>>>>>>>>> possible future work. Were there any recent discussions on >>> these >>>>>>>>>>> subjects? >>>>>>>>>>> No recent discussions on these subjects, but I think it'll >> come >>>> in >>>>>>>>>>> Flink 1.17 >>>>>>>>>>> >>>>>>>>>>>> Another related topic: are there ideas around making SQL >>>> Gateway a >>>>>>>>>>> multi-tenant >>>>>>>>>>> component? >>>>>>>>>>> Yes. >>>>>>>>>>> >>>>>>>>>>> Shengkaiis the maintainer of SQL Client and SQL gateway, >> maybe >>>> he can >>>>>>>>>>> provide more information. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Yuxia >>>>>>>>>>> >>>>>>>>>>> ----- 原始邮件 ----- >>>>>>>>>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com >>> <mailto: >>>> vendrov...@gmail.com>> >>>>>>>>>>> 收件人: "dev" <dev@flink.apache.org <mailto: >> dev@flink.apache.org >>>>> >>>>>>>>>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 >>>>>>>>>>> 主题: SQL Gateway and SQL Client >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I’m Alexey from Confluent. This is my first email in this >>>> discussion >>>>>>>>>>> list. >>>>>>>>>>> I’m rather new to Flink, and to local customs of >> communication. >>>> I want >>>>>>>>>>> to >>>>>>>>>>> dive deeper and hopefully get more involved over time. >>>>>>>>>>> >>>>>>>>>>> Currently I have a few questions around SQL Gateway and SQL >>>> Client. >>>>>>>>>>> Specifically I wanted to learn what is the vision around the >>>> nearest >>>>>>>>>>> future >>>>>>>>>>> of these two components. >>>>>>>>>>> >>>>>>>>>>> In what Flink’s release the connection from SQL Client to the >>>> Gateway is >>>>>>>>>>> expected to be added? I was looking at >>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-29486, and >>> recently >>>> it got >>>>>>>>>>> renamed from “Enable SQL Client to Connect SQL Gateway in >>> Remote >>>> Mode” >>>>>>>>>>> to >>>>>>>>>>> “Introduce Client Parser to get statement type”. I did some >>>> search, but >>>>>>>>>>> didn’t find a good place where the client's work in this >>>> direction is >>>>>>>>>>> discussed or tracked. >>>>>>>>>>> >>>>>>>>>>> A couple questions about the SQL Gateway. The FLIP-91 >>>>>>>>>>> < >>>>>>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework >>>>>>>>>>>> >>>>>>>>>>> mentions “Authentication module” (2) and “Persistent Gateway” >>>> (4) as >>>>>>>>>>> possible future work. Were there any recent discussions on >>> these >>>>>>>>>>> subjects? >>>>>>>>>>> Or maybe there are some ideas how to move these directions >>>> forward? >>>>>>>>>>> Another >>>>>>>>>>> related topic: are there ideas around making SQL Gateway a >>>> multi-tenant >>>>>>>>>>> component? >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Alexey >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>>> >>> >>
