Hi Shengkai, Regarding #2, maybe the -f options in flink and hive have different implications, and we should clarify the behavior. For example, if the client just submits the job and exits, what happens if the file contains two INSERT statements? I don't think we should treat them as a statement set, because users should explicitly write BEGIN STATEMENT SET in that case. And the client shouldn't asynchronously submit the two jobs, because the 2nd may depend on the 1st, right?
On Fri, Jan 29, 2021 at 4:30 PM Shengkai Fang <fskm...@gmail.com> wrote: > Hi Rui, > Thanks for your feedback. I agree with your suggestions. > > For the suggestion 1: Yes. we are plan to strengthen the set command. In > the implementation, it will just put the key-value into the > `Configuration`, which will be used to generate the table config. If hive > supports to read the setting from the table config, users are able to set > the hive-related settings. > > For the suggestion 2: The -f parameter will submit the job and exit. If > the queries never end, users have to cancel the job by themselves, which is > not reliable(people may forget their jobs). In most case, queries are used > to analyze the data. Users should use queries in the interactive mode. > > Best, > Shengkai > > Rui Li <lirui.fu...@gmail.com> 于2021年1月29日周五 下午3:18写道： > >> Thanks Shengkai for bringing up this discussion. I think it covers a lot >> of useful features which will dramatically improve the usability of our SQL >> Client. I have two questions regarding the FLIP. >> >> 1. Do you think we can let users set arbitrary configurations via the SET >> command? A connector may have its own configurations and we don't have a >> way to dynamically change such configurations in SQL Client. For example, >> users may want to be able to change hive conf when using hive connector [1]. >> 2. Any reason why we have to forbid queries in SQL files specified with >> the -f option? Hive supports a similar -f option but allows queries in the >> file. And a common use case is to run some query and redirect the results >> to a file. So I think maybe flink users would like to do the same, >> especially in batch scenarios. >> >> [1] https://issues.apache.org/jira/browse/FLINK-20590 >> >> On Fri, Jan 29, 2021 at 10:46 AM Sebastian Liu <liuyang0...@gmail.com> >> wrote: >> >>> Hi Shengkai, >>> >>> Glad to see this improvement. And I have some additional suggestions: >>> >>> #1. Unify the TableEnvironment in ExecutionContext to >>> StreamTableEnvironment for both streaming and batch sql. >>> #2. Improve the way of results retrieval: sql client collect the results >>> locally all at once using accumulators at present, >>> which may have memory issues in JM or Local for the big query >>> result. >>> Accumulator is only suitable for testing purpose. >>> We may change to use SelectTableSink, which is based >>> on CollectSinkOperatorCoordinator. >>> #3. Do we need to consider Flink SQL gateway which is in FLIP-91. Seems >>> that this FLIP has not moved forward for a long time. >>> Provide a long running service out of the box to facilitate the sql >>> submission is necessary. >>> >>> What do you think of these? >>> >>> [1] >>> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway >>> >>> >>> Shengkai Fang <fskm...@gmail.com> 于2021年1月28日周四 下午8:54写道： >>> >>> > Hi devs, >>> > >>> > Jark and I want to start a discussion about FLIP-163:SQL Client >>> > Improvements. >>> > >>> > Many users have complained about the problems of the sql client. For >>> > example, users can not register the table proposed by FLIP-95. >>> > >>> > The main changes in this FLIP: >>> > >>> > - use -i parameter to specify the sql file to initialize the table >>> > environment and deprecated YAML file; >>> > - add -f to submit sql file and deprecated '-u' parameter; >>> > - add more interactive commands, e.g ADD JAR; >>> > - support statement set syntax; >>> > >>> > >>> > For more detailed changes, please refer to FLIP-163[1]. >>> > >>> > Look forward to your feedback. >>> > >>> > >>> > Best, >>> > Shengkai >>> > >>> > [1] >>> > >>> > >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-163%3A+SQL+Client+Improvements >>> > >>> >>> >>> -- >>> >>> *With kind regards >>> ------------------------------------------------------------ >>> Sebastian Liu 刘洋 >>> Institute of Computing Technology, Chinese Academy of Science >>> Mobile\WeChat: +86—15201613655 >>> E-mail: liuyang0...@gmail.com <liuyang0...@gmail.com> >>> QQ: 3239559* >>> >> >> >> -- >> Best regards! >> Rui Li >> > -- Best regards! Rui Li
