+1 to fix it in 1.10. If this feature doesn't work via SQL CLI, I guess it doesn't work for most Hive users. +0 to remove the dialect check. I don't see much benefit this check can bring to users, except that it prevents users from accidentally using some Hive features, which doesn't seem to be very harmful anyway. But I don't have a strong opinion for it and it probably needs more thorough discussion.
On Thu, Dec 12, 2019 at 7:49 AM Jark Wu <imj...@gmail.com> wrote: > Thanks Jingsong, > > OVERWRITE and PARTITION are very fundamental features for Hive users. > I'm sorry to hear that it doesn't work in SQL CLI. > > > Remove hive dialect limitation for these two grammars? > The dialect restriction is introduced on purpose, because OVERWRITE and > PARTITION syntax > are not SQL standard. Even in the discussion of FLIP-63, the community > have different opinion on > whether the partition fields should be declared in schema part or not [1]. > The status of FLIP-63 is > that limit the proposed syntax in Hive dialect, and we may propose new > built-in syntax for creating > partition tables in the future. So I'm -1 to removing dialect limitation. > > > Should we fix this in 1.10? > From my point of view, the problem is that users can't switch dialects in > SQL CLI, because it is not > exposed as a configuration. FLIP-89 [2] proposed to make TableConfig > configurable including the > dialect configuration, i.g. `table.planner.sql-dialect`. So the only > thing we need to do is introducing > such a configuration, but this is definitely is not a *bug*. However, > considering it is a small change, > and from user perspective, I'm +1 to introducing such a configuration. > > Best, > Jark > > [1]: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html > [2]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig > > On Thu, 12 Dec 2019 at 03:24, Bowen Li <bowenl...@gmail.com> wrote: > >> Hi Jingsong, >> >> Thanks a lot for reporting this issue. >> >> IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support Hive >> integration before FLIP-63 was proposed to introduce generic partition >> support to Flink. Thus when we added these syntax, we were intentionally >> conservative and limited their scope to Hive dialect. @Rui may help to >> confirm that. I'm a bit surprised about it as well. As core APIs of >> FLIP-63 >> were done, I don't see why we would limit these syntax to Hive dialect >> alone. It's just unfortunately that we may have forgot to revisit this >> topic and we apparently missed some test cases on SQL CLI side. Sorry for >> that. >> >> From a product perspective, SQL CLI is super critical for Flink-Hive >> integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are two >> of the most commonly used syntax in Hive, the product wouldn't be useable >> without such proper support. >> >> Thus, I think this is a *bug*, we can stop limiting them in Hive dialect, >> and should fix it in 1.10. We should also add more test coverage for SQL >> CLI to avoid such surprise. >> >> Cheers, >> Bowen >> >> On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li <jingsongl...@gmail.com> >> wrote: >> >> > Hi Dev, >> > >> > After cutting out the branch of 1.10, I tried the following functions of >> > SQL-CLI and found that it does not support: >> > - insert overwrite >> > - PARTITION (partcol1=val1, partcol2=val2 ...) >> > The SQL pattern is: >> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1, >> > partcol2=val2 ...) select_statement1 FROM from_statement; >> > It is a surprise to me. >> > The reason is that we only allow these two grammars in hive dialect. And >> > SQL-CLI does not have an interface to switch dialects. >> > >> > Because it directly hinders the SQL-CLI's insert syntax in hive >> integration >> > and seriously hinders the practicability of SQL-CLI. >> > And we have introduced these two grammars in FLIP-63 [1] to Flink. >> > Here are my question: >> > 1.Should we remove hive dialect limitation for these two grammars? >> > 2.Should we fix this in 1.10? >> > >> > [1] >> > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support >> > >> > Best, >> > Jingsong Lee >> > >> > -- Best regards! Rui Li
