Thanks Jingsong, OVERWRITE and PARTITION are very fundamental features for Hive users. I'm sorry to hear that it doesn't work in SQL CLI.
> Remove hive dialect limitation for these two grammars? The dialect restriction is introduced on purpose, because OVERWRITE and PARTITION syntax are not SQL standard. Even in the discussion of FLIP-63, the community have different opinion on whether the partition fields should be declared in schema part or not [1]. The status of FLIP-63 is that limit the proposed syntax in Hive dialect, and we may propose new built-in syntax for creating partition tables in the future. So I'm -1 to removing dialect limitation. > Should we fix this in 1.10? >From my point of view, the problem is that users can't switch dialects in SQL CLI, because it is not exposed as a configuration. FLIP-89 [2] proposed to make TableConfig configurable including the dialect configuration, i.g. `table.planner.sql-dialect`. So the only thing we need to do is introducing such a configuration, but this is definitely is not a *bug*. However, considering it is a small change, and from user perspective, I'm +1 to introducing such a configuration. Best, Jark [1]: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig On Thu, 12 Dec 2019 at 03:24, Bowen Li <bowenl...@gmail.com> wrote: > Hi Jingsong, > > Thanks a lot for reporting this issue. > > IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support Hive > integration before FLIP-63 was proposed to introduce generic partition > support to Flink. Thus when we added these syntax, we were intentionally > conservative and limited their scope to Hive dialect. @Rui may help to > confirm that. I'm a bit surprised about it as well. As core APIs of FLIP-63 > were done, I don't see why we would limit these syntax to Hive dialect > alone. It's just unfortunately that we may have forgot to revisit this > topic and we apparently missed some test cases on SQL CLI side. Sorry for > that. > > From a product perspective, SQL CLI is super critical for Flink-Hive > integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are two > of the most commonly used syntax in Hive, the product wouldn't be useable > without such proper support. > > Thus, I think this is a *bug*, we can stop limiting them in Hive dialect, > and should fix it in 1.10. We should also add more test coverage for SQL > CLI to avoid such surprise. > > Cheers, > Bowen > > On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li <jingsongl...@gmail.com> > wrote: > > > Hi Dev, > > > > After cutting out the branch of 1.10, I tried the following functions of > > SQL-CLI and found that it does not support: > > - insert overwrite > > - PARTITION (partcol1=val1, partcol2=val2 ...) > > The SQL pattern is: > > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1, > > partcol2=val2 ...) select_statement1 FROM from_statement; > > It is a surprise to me. > > The reason is that we only allow these two grammars in hive dialect. And > > SQL-CLI does not have an interface to switch dialects. > > > > Because it directly hinders the SQL-CLI's insert syntax in hive > integration > > and seriously hinders the practicability of SQL-CLI. > > And we have introduced these two grammars in FLIP-63 [1] to Flink. > > Here are my question: > > 1.Should we remove hive dialect limitation for these two grammars? > > 2.Should we fix this in 1.10? > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > > > Best, > > Jingsong Lee > > >
