Hi Timo, I understand we need further discussion about syntax/dialect for 1.11. But as Jark has pointed out, the current implementation violates the accepted design of FLIP-63, which IMO qualifies as a bug. Given that it's a bug and has great impact on the usability of our Hive integration, do you think we can fix it in 1.10?
On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li <jingsongl...@gmail.com> wrote: > Hi Timo, > > I am OK if you think they are not bug and they should not be included in > 1.10. > > I think they have been accepted in FLIP-63. And there is no objection. It > has been more than three months since the discussion of FLIP-63. It's been > six months since Flink added these two syntaxs. > > But I can also start discussion and vote thread for FLIP-63 again, to make > sure once again that everyone is happy. > > Best, > Jingsong Lee > > On Thu, Dec 12, 2019 at 11:35 PM Timo Walther <twal...@apache.org> wrote: > > > Hi Jingsong, > > > > I will also add my opinion here for future discussions: > > > > We had long discussions around SQL syntax in the past (e.g. for > > WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the > > end all parties were happy and we came up with a good long-term solution > > that is unlikely to be changed in the near future. IMHO we can differ > > from the standard esp. for DDL statements (every SQL vendors has custom > > syntax there) but still we should have time to hear many opinions. > > However, for DQL and DML statements we need to be even more cautious > > because here we enter SQL standard areas. > > > > Happy to discuss a partion syntax for 1.11 and go with the solution that > > Jark proposed using a config option. > > > > Thanks, > > Timo > > > > > > On 12.12.19 09:40, Jark Wu wrote: > > > Hi Jingsong, > > > > > > Thanks for the explanation, I think I misunderstood your point at the > > > begining. > > > As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are > > added > > > to Flink's > > > SQL syntax, but CREATE PARTITION TABLE should be limited under Hive > > > dialect. > > > However, the current implementation is opposite, INSERT OVERWRITE > > > and INSERT PARTITION are > > > under the dialect limitation, but CREATE PARTITION TABLE is not. > > > > > > So it is indeed a bug which should be fixed in 1.10. > > > > > > Best, > > > Jark > > > > > > On Thu, 12 Dec 2019 at 16:35, Jingsong Li <jingsongl...@gmail.com> > > wrote: > > > > > >> Hi Jark, > > >> > > >> Let's recall FLIP-63, > > >> We supported these syntax in hive dialect at 1.9. All of my reasons > for > > >> launching FLIP-63 are to bring partition support to Flink itself. > > >> Not only batch, but also we have the need to stream jobs to write > > partition > > >> files today, which is also one of our very important application > > scenarios. > > >> > > >> The original intention of FLIP-63 is to bring all partition syntax to > > >> Flink, but in the end you and I have some different opinion in > creating > > >> partition table, so our consensus is to leave it in hive dialect, only > > it. > > >> > > >> [1] > > >> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > >> > > >> Best, > > >> Jingsong Lee > > >> > > >> On Thu, Dec 12, 2019 at 4:05 PM Jark Wu <imj...@gmail.com> wrote: > > >> > > >>> Hi jingsong, > > >>> > > >>> Watermark is not a standard syntax, that's why we had a FLIP and long > > >>> discussion to add it to > > >>> Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and > > >>> PARTITION syntax to > > >>> Flink's own syntax, we also need a FLIP or a VOTE, and this may > > can't > > >>> happen soon (we should > > >>> hear more people's opinions on this). > > >>> > > >>> Regarding to the sql-dialect configuration, I was not saying to > involve > > >> the > > >>> whole FLIP-89. I mean we can just > > >>> start a VOTE to expose it as `table.planner.sql-dialect` and include > it > > >> in > > >>> 1.10. The change can be very small, by > > >>> adding a ConfigOption and changing the implementation of > > >>> TableConfig#getSqlDialect/setSqlDialect. I believe > > >>> it is smaller and safer than changing the parser. > > >>> > > >>> Btw, I cc'ed Yu Li and Gary into the discussion, because release > > managers > > >>> should be aware of this. > > >>> > > >>> Best, > > >>> Jark > > >>> > > >>> > > >>> > > >>> On Thu, 12 Dec 2019 at 11:47, Danny Chan <yuzhao....@gmail.com> > wrote: > > >>> > > >>>> Thanks Jingsong for bring up this discussion ~ > > >>>> > > >>>> After reviewing FLIP-63, it seems that we have made a conclusion for > > >> the > > >>>> syntax > > >>>> > > >>>> - INSERT OVERWRITE ... > > >>>> - INSERT INTO … PARTITION > > >>>> > > >>>> Which means that they should not have the Hive dialect limitation, > so > > >> I’m > > >>>> inclined that the behaviors for SQL-CLI is unexpected, or a “bug” > that > > >>> need > > >>>> to fix. > > >>>> > > >>>> We did not make a conclusion for the syntax: > > >>>> > > >>>> - CREATE TABLE … PARTITIONED BY ... > > >>>> > > >>>> Which means that the behavior of it is under-discussion, so it is > okey > > >> to > > >>>> be without the HIVE dialect limitation, we do not actually have any > > >> table > > >>>> sources/sinks that support such a DDL so for current code base, > users > > >>>> should not be influenced by the behaviors change. > > >>>> > > >>>> So I’m > > >>>> > > >>>> +1 to remove the hive dialect limitations for INSERT OVERWRITE and > > >> INSERT > > >>>> PARTITION > > >>>> +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not > finished > > >>>> yet, we better do this until FLIP-89 is resolved. > > >>>> > > >>>> Best, > > >>>> Danny Chan > > >>>> 在 2019年12月11日 +0800 PM5:29，Jingsong Li <jingsongl...@gmail.com>，写道： > > >>>>> Hi Dev, > > >>>>> > > >>>>> After cutting out the branch of 1.10, I tried the following > functions > > >>> of > > >>>>> SQL-CLI and found that it does not support: > > >>>>> - insert overwrite > > >>>>> - PARTITION (partcol1=val1, partcol2=val2 ...) > > >>>>> The SQL pattern is: > > >>>>> INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION > > >> (partcol1=val1, > > >>>>> partcol2=val2 ...) select_statement1 FROM from_statement; > > >>>>> It is a surprise to me. > > >>>>> The reason is that we only allow these two grammars in hive > dialect. > > >>> And > > >>>>> SQL-CLI does not have an interface to switch dialects. > > >>>>> > > >>>>> Because it directly hinders the SQL-CLI's insert syntax in hive > > >>>> integration > > >>>>> and seriously hinders the practicability of SQL-CLI. > > >>>>> And we have introduced these two grammars in FLIP-63 [1] to Flink. > > >>>>> Here are my question: > > >>>>> 1.Should we remove hive dialect limitation for these two grammars? > > >>>>> 2.Should we fix this in 1.10? > > >>>>> > > >>>>> [1] > > >>>>> > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > >>>>> > > >>>>> Best, > > >>>>> Jingsong Lee > > >>>> > > >>> > > >> > > >> > > >> -- > > >> Best, Jingsong Lee > > >> > > > > > > > > > -- > Best, Jingsong Lee > -- Best regards! Rui Li
