LI easy to support this configuration by
passing
through to the TableEnv.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-163-SQL-Client-Improvements-tp48354p48665.html
On Wed, 24 Feb 2021 at 10:39, Kurt Young
wrote:
If we all agree t
t portability. Declaring the
> > > sync/async
> > > >>>> behavior will happen in many SQL scripts. And users should be
> easily
> > > >>>> switch from SQL Client to some commercial product without the need
> > of
> > > >>&
t; > >>>>
> > >>>> Sure, we can change from `sql-client.dml-sync` to `table.dml-sync`
> > later
> > >>>> but that would mean introducing future confusion. An app name (what
> > >>>> `sql-client` kind of is) should not be part of a conf
part of a config option key if
> >>>> other apps will need the same kind of option.
> >>>>
> >>>> Regards,
> >>>> Timo
> >>>>
> >>>>
> >>>> On 24.02.21 08:59, Jark Wu wrote:
> >>&
e TableEnv.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-163-SQL-Client-Improvements-tp48354p48665.html
On Wed, 24 Feb 2021 at 10:39, Kurt Young wrote:
If we all agree the option should only be handled by sql client, then
why
don't we
ju
>>> If we want to introduce an unified "table.dml-sync" option, I prefer
> >>> it should be implemented on Table API and affect all the DMLs on
> >>> Table API (`tEnv.executeSql`, `Table.executeInsert`, `StatementSet`),
> >>> as I have mentioned
before [1].
>>>
>>>> It would be very straightforward that it affects all the DMLs on SQL CLI
>>> and
>>> TableEnvironment (including `executeSql`, `StatementSet`,
>>> `Table#executeInsert`, etc.).
>>> This can also make SQL CLI easy to support th
, etc.).
> > This can also make SQL CLI easy to support this configuration by passing
> > through to the TableEnv.
> >
> > Best,
> > Jark
> >
> >
> > [1]:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-1
ronment (including `executeSql`, `StatementSet`,
`Table#executeInsert`, etc.).
This can also make SQL CLI easy to support this configuration by passing
through to the TableEnv.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-163-SQL-Client-Improvements-tp483
through to the TableEnv.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-163-SQL-Client-Improvements-tp48354p48665.html
On Wed, 24 Feb 2021 at 10:39, Kurt Young wrote:
> If we all agree the option should only be handled by sql client, then why
&g
If we all agree the option should only be handled by sql client, then why
don't we
just call it `sql-client.dml-sync`? As you said, calling it
`table.dml-sync` but has no
affection in `TableEnv.executeSql("INSERT INTO")` will also cause a big
confusion for
users.
The only concern I saw is if we
Hi Kurt,
we can also shorten it to `table.dml-sync` if that would help. Then it
would confuse users that do a regular `.executeSql("INSERT INTO")` in a
notebook session.
In any case users will need to learn the semantics of this option.
`table.multi-dml-sync` should be described as "If a
Sorry for the late reply, but I'm confused by `table.multi-dml-sync`.
IIUC this config will take effect with 2 use cases:
1. SQL client, either interactive mode or executing multiple statements via
-f. In most cases,
there will be only one INSERT INTO statement but we are controlling the
Hi everyone.
Sorry for the late response.
For `execution.runtime-mode`, I think it's much better than
`table.execution.mode`. Thanks for Timo's suggestions!
For `SHOW CREATE TABLE`, I'm +1 with Jark's comments. We should clarify the
usage of the SHOW CREATE TABLE statements. It should be
Hi Ingo,
1) I think you are right, the table path should be fully-qualified.
2) I think this is also a good point. The SHOW CREATE TABLE
only aims to print DDL for the tables registered using SQL CREATE TABLE
DDL.
If a table is registered using Table API, e.g.
Hi all,
I have a couple questions about the SHOW CREATE TABLE statement.
1) Contrary to the example in the FLIP I think the returned DDL should
always have the table identifier fully-qualified. Otherwise the DDL depends
on the current context (catalog/database), which could be surprising,
Hi Shengkai,
thanks for updating the FLIP.
I have one last comment for the option `table.execution.mode`. Should we
already use the global Flink option `execution.runtime-mode` instead?
We are using Flink's options where possible (e.g. `pipeline.name` and
`parallism.default`) why not also
Hi, guys.
I have updated the FLIP. It seems we have reached agreement. Maybe we can
start the vote soon. If anyone has other questions, please leave your
comments.
Best，
Shengkai
Rui Li 于2021年2月9日 周二下午7:52写道：
> Hi guys,
>
> The conclusion sounds good to me.
>
> On Tue, Feb 9, 2021 at 5:39 PM
Hi guys,
The conclusion sounds good to me.
On Tue, Feb 9, 2021 at 5:39 PM Shengkai Fang wrote:
> Hi, Timo, Jark.
>
> I am fine with the new option name.
>
> Best,
> Shengkai
>
> Timo Walther 于2021年2月9日 周二下午5:35写道：
>
> > Yes, `TableEnvironment#executeMultiSql()` can be future work.
> >
> >
Hi, Timo, Jark.
I am fine with the new option name.
Best,
Shengkai
Timo Walther 于2021年2月9日 周二下午5:35写道：
> Yes, `TableEnvironment#executeMultiSql()` can be future work.
>
> @Rui, Shengkai: Are you also fine with this conclusion?
>
> Thanks,
> Timo
>
> On 09.02.21 10:14, Jark Wu wrote:
> > I'm
Yes, `TableEnvironment#executeMultiSql()` can be future work.
@Rui, Shengkai: Are you also fine with this conclusion?
Thanks,
Timo
On 09.02.21 10:14, Jark Wu wrote:
I'm fine with `table.multi-dml-sync`.
My previous concern about "multi" is that DML in CLI looks like single
statement.
But we
I'm fine with `table.multi-dml-sync`.
My previous concern about "multi" is that DML in CLI looks like single
statement.
But we can treat CLI as a multi-line accepting statements from opening to
closing.
Thus, I'm fine with `table.multi-dml-sync`.
So the conclusion is `table.multi-dml-sync`
Hi everyone,
I understand Rui's concerns. `table.dml-sync` should not apply to
regular `executeSql`. Actually, this option makes only sense when
executing multi statements. Once we have a
`TableEnvironment.executeMultiSql()` this config could be considered.
Maybe we can find a better
Hi, all.
I think it may cause user confused. The main problem is we have no means
to detect the conflict configuration, e.g. users set the option true and
use `TableResult#await` together.
Best,
Shengkai.
Hi Jark,
I agree it's more consistent if table API also respects this config. But on
the other hand, it might make the `executeSql` API a little trickier to
use, because now DDL, DQL and DML all behave differently from one another:
- DDL: always sync
- DQL: always async
- DML: can be
Hi Rui,
That's a good point. From the naming of the option, I prefer to get sync
behavior.
It would be very straightforward that it affects all the DMLs on SQL CLI
and
TableEnvironment (including `executeSql`, `StatementSet`,
`Table#executeInsert`, etc.).
This can also make SQL CLI easy to
Hi,
Glad to see we have reached consensus on option #2. +1 to it.
Regarding the name, I'm fine with `table.dml-async`. But I wonder whether
this config also applies to table API. E.g. if a user
sets table.dml-async=false and calls TableEnvironment::executeSql to run a
DML, will he get sync
Ah, I just forgot the option name.
I'm also fine with `table.dml-async`.
What do you think @Rui Li @Shengkai Fang
?
Best,
Jark
On Mon, 8 Feb 2021 at 23:06, Timo Walther wrote:
> Great to hear that. Can someone update the FLIP a final time before we
> start a vote?
>
> We should quickly
Great to hear that. Can someone update the FLIP a final time before we
start a vote?
We should quickly discuss how we would like to name the config option
for the async/sync mode. I heared voices internally that are strongly
against calling it "detach" due to historical reasons with a Flink
Thanks Timo,
I'm +1 for option#2 too.
I think we have addressed all the concerns and can start a vote.
Best,
Jark
On Mon, 8 Feb 2021 at 22:19, Timo Walther wrote:
> Hi Jark,
>
> you are right. Nesting STATEMENT SET and ASYNC might be too verbose.
>
> So let's stick to the config option
Hi Jark,
you are right. Nesting STATEMENT SET and ASYNC might be too verbose.
So let's stick to the config option approach.
However, I strongly believe that we should not use the batch/streaming
mode for deriving semantics. This discussion is similar to time function
discussion. We should
Hi Timo,
Actually, I'm not in favor of explicit syntax `BEGIN ASYNC;... END;`.
Because it makes submitting streaming jobs very verbose, every INSERT INTO
and STATEMENT SET must be wrapped in the ASYNC clause which is
not user-friendly and not backward-compatible.
I agree we will have unified
Hi Jark, Hi Rui,
1) How should we execute statements in CLI and in file? Should there be
a difference?
So it seems we have consensus here with unified bahavior. Even though
this means we are breaking existing batch INSERT INTOs that were
asynchronous before.
2) Should we have different
Hi Timo,
I agree with Jark that we should provide consistent experience regarding
SQL CLI and files. Some systems even allow users to execute SQL files in
the CLI, e.g. the "SOURCE" command in MySQL. If we want to support that in
the future, it's a little tricky to decide whether that should be
Hi Timo,
1) How should we execute statements in CLI and in file? Should there be a
difference?
I do think we should unify the behavior of CLI and SQL files. SQL files can
be thought of as a shortcut of
"start CLI" => "copy content of SQL files" => "past content in CLI".
Actually, we already did
Hi Jark,
thanks for the summary. I hope we can also find a good long-term
solution on the async/sync execution behavior topic.
It should be discussed in a bigger round because it is (similar to the
time function discussion) related to batch-streaming unification where
we should stick to the
Hi all,
After an offline discussion with Timo and Kurt, we have reached some
consensus.
Please correct me if I am wrong or missed anything.
1) We will introduce "table.planner" and "table.execution-mode" instead of
"sql-client" prefix,
and add `TableEnvironment.create(Configuration)` interface.
Hi Ingo,
Since we have supported the WITH syntax and SET command since v1.9 [1][2],
and
we have never received such complaints, I think it's fine for such
differences.
Besides, the TBLPROPERTIES clause of CREATE TABLE in Hive also requires
string literal keys[3],
and the SET = doesn't allow
Hi,
regarding the (un-)quoted question, compatibility is of course an important
argument, but in terms of consistency I'd find it a bit surprising that
WITH handles it differently than SET, and I wonder if that could cause
friction for developers when writing their SQL.
Regards
Ingo
On Thu,
Hi all,
Regarding "One Parser", I think it's not possible for now because Calcite
parser can't parse
special characters (e.g. "-") unless quoting them as string literals.
That's why the WITH option
key are string literals not identifiers.
SET table.exec.mini-batch.enabled = true and ADD JAR
Hi everyone,
Regarding "table.planner" and "table.execution-mode"
If we define that those two options are just used to initialize the
TableEnvironment, +1 for introducing table options instead of sql-client
options.
Regarding "the sql client, we will maintain two parsers", I want to give
more
Hi guys,
Regarding #3 and #4, I agree SHOW JARS is more consistent with other
commands than LIST JARS. I don't have a strong opinion about REMOVE vs
DELETE though.
While flink doesn't need to follow hive syntax, as far as I know, most
users who are requesting these features are previously hive
Hi everyone,
some feedback regarding the open questions. Maybe we can discuss the
`TableEnvironment.executeMultiSql` story offline to determine how we
proceed with this in the near future.
1) "whether the table environment has the ability to update itself"
Maybe there was some
Hi Timo,
I will respond some of the questions:
1) SQL client specific options
Whether it starts with "table" or "sql-client" depends on where the
configuration takes effect.
If it is a table configuration, we should make clear what's the behavior
when users change
the configuration in the
Hi, Timo.
Thanks for your detailed feedback. I have some thoughts about your feedback.
*Regarding #1*: I think the main problem is whether the table environment
has the ability to update itself. Let's take a simple program as an example.
```
TableEnvironment tEnv = TableEnvironment.create(...);
Thanks for this great proposal Shengkai. This will give the SQL Client a
very good update and make it production ready.
Here is some feedback from my side:
1) SQL client specific options
I don't think that `sql-client.planner` and `sql-client.execution.mode`
are SQL Client specific. Similar
Sorry for the typo. I mean `RESET` is much better rather than `UNSET`.
Shengkai Fang 于2021年2月2日周二 下午4:44写道：
> Hi, Jingsong.
>
> Thanks for your reply. I think `UNSET` is much better.
>
> 1. We don't need to introduce another command `UNSET`. `RESET` is
> supported in the current sql client now.
Hi, Jingsong.
Thanks for your reply. I think `UNSET` is much better.
1. We don't need to introduce another command `UNSET`. `RESET` is supported
in the current sql client now. Our proposal just extends its grammar and
allow users to reset the specified keys.
2. Hive beeline also uses `RESET` to
Thanks for the proposal, yes, sql-client is too outdated. +1 for improving
it.
About "SET" and "RESET", Why not be "SET" and "UNSET"?
Best,
Jingsong
On Mon, Feb 1, 2021 at 2:46 PM Rui Li wrote:
> Thanks Shengkai for the update! The proposed changes look good to me.
>
> On Fri, Jan 29, 2021
Thanks Shengkai for the update! The proposed changes look good to me.
On Fri, Jan 29, 2021 at 8:26 PM Shengkai Fang wrote:
> Hi, Rui.
> You are right. I have already modified the FLIP.
>
> The main changes:
>
> # -f parameter has no restriction about the statement type.
> Sometimes, users use
Hi, Rui.
You are right. I have already modified the FLIP.
The main changes:
# -f parameter has no restriction about the statement type.
Sometimes, users use the pipe to redirect the result of queries to debug
when submitting job by -f parameter. It's much convenient comparing to
writing INSERT
Hi Shengkai,
Regarding #2, maybe the -f options in flink and hive have different
implications, and we should clarify the behavior. For example, if the
client just submits the job and exits, what happens if the file contains
two INSERT statements? I don't think we should treat them as a statement
Hi Rui,
Thanks for your feedback. I agree with your suggestions.
For the suggestion 1: Yes. we are plan to strengthen the set command. In
the implementation, it will just put the key-value into the
`Configuration`, which will be used to generate the table config. If hive
supports to read the
Thanks Shengkai for bringing up this discussion. I think it covers a lot of
useful features which will dramatically improve the usability of our SQL
Client. I have two questions regarding the FLIP.
1. Do you think we can let users set arbitrary configurations via the SET
command? A connector may
-- Forwarded message -
发件人： Shengkai Fang
Date: 2021年1月29日周五 下午2:46
Subject: Re: [DISCUSS]FLIP-163: SQL Client Improvements
To: Sebastian Liu
Hi Sebastian,
Thanks for your suggestions.
For the suggestion 1, 2:
Yes. That's what we want. By the way, Godfrey has already the PR
Hi Shengkai,
Glad to see this improvement. And I have some additional suggestions:
#1. Unify the TableEnvironment in ExecutionContext to
StreamTableEnvironment for both streaming and batch sql.
#2. Improve the way of results retrieval: sql client collect the results
locally all at once using
Hi devs,
Jark and I want to start a discussion about FLIP-163:SQL Client
Improvements.
Many users have complained about the problems of the sql client. For
example, users can not register the table proposed by FLIP-95.
The main changes in this FLIP:
- use -i parameter to specify the sql file
57 matches
Mail list logo
