Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Thanks all, As your suggestion, I'd like to do: - FLIP-63: move "CREATE TABLE ... PARTITIONED BY" and "MSCK REPAIR TABLE" to further discussion chapter. - Remove hive dialect limitation for supported "INSERT OVERWRITE" and "INSERT ... PARTITION(...)". - Limit "CREATE TABLE ... PARTITIONED BY" to hive dialect. Thanks everyone again. Best, Jingsong Lee On Sat, Dec 14, 2019 at 2:41 AM Xuefu Z wrote: > Thanks all for the healthy discussions. I'd just like to point out a light > difference between standard and standard compatibility. Most of DB vendors > meant the latter when they claim following a sql standard. However, it > doesn't mean they don't have any syntax beyond the standard grammar. > Extension is very common. > > Therefore, I think Flink SQL can also have more freedom in extending the > standard grammar and adopting non-standard grammars such as "INSERT > OVERWRITE" that comes from Hive. While Hive has a lot of good cases like > this, we are also free to reject or put for a specific dialect anything > that's too specific or doesn't make sense in Flink. To me, "NSERT > OVERWRITE" can be adopted, which "MSCK REPAIR TABLE" can be rejected or put > as "Hive dailect". > > Lastly, if FLIP is officially accepted, I agree that we should respect and > fix things as bugs accordingly. If there is a second thought or debate on > anything, a vote is required to overturn it. Before that, the original FLIP > holds. > > Thanks, > Xuefu > > > On Fri, Dec 13, 2019 at 1:20 AM Jingsong Li > wrote: > > > Hi Timo, > > > > Thanks for your feedback. > > > > The reason of `The DDL can like this (With hive dialect)` is: > > The syntax of creating partition table is controversial, so we think we > > should put it aside for the time being to make it invisible to users. > Since > > we implemented this syntax in 1.9, we decided to put it under hive > dialect. > > (Subsequent votes will be taken to determine.) > > MSCK means metastore check, It is listed in the document mainly for the > > integrity of the story. Of course, it can be ignored. I fully agree that > it > > can be discussed slowly in the future. > > > > >The current problem for the CLI is that it still does not use the > > flink-sql-parser. We can support all new syntax in Flink 1.11 once the > CLI > > architecture has changed there. With the current architecture (using > simple > > regex), I'm not sure if we can support all the grammar listed in the > FLIP. > > > > You are right, we can not support any DDL grammars in FLIP. we should > wait > > refactor of SQL-CLI in 1.11. > > What we can do is to remove the restriction of hiveDialect for the insert > > (overwrite and partition) syntax. That's what we've supported, and so far > > we haven't received any objections. > > > > Best, > > Jingsong Lee > > > > On Fri, Dec 13, 2019 at 4:51 PM Timo Walther wrote: > > > > > Hi everyone, > > > > > > sorry, I was not aware that FLIP-63 already lists a lot of additional > > > SQL grammar. It was accepted though an official voting process so I > > > guess we can adopt the listed grammar for Flink SQL. > > > > > > The only thing that confuses me is the mentioning of `The DDL can like > > > this (With hive dialect)`, for the next time we should make it more > > > explicit what belongs to the Flink SQL dialect and what belongs to the > > > Hive dialect. Also the not intuitive `MSCK REPAIR TABLE` seems strange > > > to me. What does MSCK stand for? Maybe we should skip stuff like that > > > for now. > > > > > > The current problem for the CLI is that it still does not use the > > > flink-sql-parser. We can support all new syntax in Flink 1.11 once the > > > CLI architecture has changed there. With the current architecture > (using > > > simple regex), I'm not sure if we can support all the grammar listed in > > > the FLIP. > > > > > > Regards, > > > Timo > > > > > > On 13.12.19 04:22, Rui Li wrote: > > > > Hi Timo, > > > > > > > > I understand we need further discussion about syntax/dialect for > 1.11. > > > But > > > > as Jark has pointed out, the current implementation violates the > > accepted > > > > design of FLIP-63, which IMO qualifies as a bug. Given that it's a > bug > > > and > > > > has great impact on the usability of our Hive integration, do you > think > > > we > > > > can fix it in 1.10? > > > > > > > > On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li > > > > wrote: > > > > > > > >> Hi Timo, > > > >> > > > >> I am OK if you think they are not bug and they should not be > included > > in > > > >> 1.10. > > > >> > > > >> I think they have been accepted in FLIP-63. And there is no > objection. > > > It > > > >> has been more than three months since the discussion of FLIP-63. > It's > > > been > > > >> six months since Flink added these two syntaxs. > > > >> > > > >> But I can also start discussion and vote thread for FLIP-63 again, > to > > > make > > > >> sure once again that everyone is happy. > > > >> > > > >> Best, > > > >> Jingsong Lee > > > >> > > > >> On Thu, D
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Thanks all for the healthy discussions. I'd just like to point out a light difference between standard and standard compatibility. Most of DB vendors meant the latter when they claim following a sql standard. However, it doesn't mean they don't have any syntax beyond the standard grammar. Extension is very common. Therefore, I think Flink SQL can also have more freedom in extending the standard grammar and adopting non-standard grammars such as "INSERT OVERWRITE" that comes from Hive. While Hive has a lot of good cases like this, we are also free to reject or put for a specific dialect anything that's too specific or doesn't make sense in Flink. To me, "NSERT OVERWRITE" can be adopted, which "MSCK REPAIR TABLE" can be rejected or put as "Hive dailect". Lastly, if FLIP is officially accepted, I agree that we should respect and fix things as bugs accordingly. If there is a second thought or debate on anything, a vote is required to overturn it. Before that, the original FLIP holds. Thanks, Xuefu On Fri, Dec 13, 2019 at 1:20 AM Jingsong Li wrote: > Hi Timo, > > Thanks for your feedback. > > The reason of `The DDL can like this (With hive dialect)` is: > The syntax of creating partition table is controversial, so we think we > should put it aside for the time being to make it invisible to users. Since > we implemented this syntax in 1.9, we decided to put it under hive dialect. > (Subsequent votes will be taken to determine.) > MSCK means metastore check, It is listed in the document mainly for the > integrity of the story. Of course, it can be ignored. I fully agree that it > can be discussed slowly in the future. > > >The current problem for the CLI is that it still does not use the > flink-sql-parser. We can support all new syntax in Flink 1.11 once the CLI > architecture has changed there. With the current architecture (using simple > regex), I'm not sure if we can support all the grammar listed in the FLIP. > > You are right, we can not support any DDL grammars in FLIP. we should wait > refactor of SQL-CLI in 1.11. > What we can do is to remove the restriction of hiveDialect for the insert > (overwrite and partition) syntax. That's what we've supported, and so far > we haven't received any objections. > > Best, > Jingsong Lee > > On Fri, Dec 13, 2019 at 4:51 PM Timo Walther wrote: > > > Hi everyone, > > > > sorry, I was not aware that FLIP-63 already lists a lot of additional > > SQL grammar. It was accepted though an official voting process so I > > guess we can adopt the listed grammar for Flink SQL. > > > > The only thing that confuses me is the mentioning of `The DDL can like > > this (With hive dialect)`, for the next time we should make it more > > explicit what belongs to the Flink SQL dialect and what belongs to the > > Hive dialect. Also the not intuitive `MSCK REPAIR TABLE` seems strange > > to me. What does MSCK stand for? Maybe we should skip stuff like that > > for now. > > > > The current problem for the CLI is that it still does not use the > > flink-sql-parser. We can support all new syntax in Flink 1.11 once the > > CLI architecture has changed there. With the current architecture (using > > simple regex), I'm not sure if we can support all the grammar listed in > > the FLIP. > > > > Regards, > > Timo > > > > On 13.12.19 04:22, Rui Li wrote: > > > Hi Timo, > > > > > > I understand we need further discussion about syntax/dialect for 1.11. > > But > > > as Jark has pointed out, the current implementation violates the > accepted > > > design of FLIP-63, which IMO qualifies as a bug. Given that it's a bug > > and > > > has great impact on the usability of our Hive integration, do you think > > we > > > can fix it in 1.10? > > > > > > On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li > > wrote: > > > > > >> Hi Timo, > > >> > > >> I am OK if you think they are not bug and they should not be included > in > > >> 1.10. > > >> > > >> I think they have been accepted in FLIP-63. And there is no objection. > > It > > >> has been more than three months since the discussion of FLIP-63. It's > > been > > >> six months since Flink added these two syntaxs. > > >> > > >> But I can also start discussion and vote thread for FLIP-63 again, to > > make > > >> sure once again that everyone is happy. > > >> > > >> Best, > > >> Jingsong Lee > > >> > > >> On Thu, Dec 12, 2019 at 11:35 PM Timo Walther > > wrote: > > >> > > >>> Hi Jingsong, > > >>> > > >>> I will also add my opinion here for future discussions: > > >>> > > >>> We had long discussions around SQL syntax in the past (e.g. for > > >>> WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in > > the > > >>> end all parties were happy and we came up with a good long-term > > solution > > >>> that is unlikely to be changed in the near future. IMHO we can differ > > >>> from the standard esp. for DDL statements (every SQL vendors has > custom > > >>> syntax there) but still we should have time to hear many opinions. > > >>> However, for DQ
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Timo, Thanks for your feedback. The reason of `The DDL can like this (With hive dialect)` is: The syntax of creating partition table is controversial, so we think we should put it aside for the time being to make it invisible to users. Since we implemented this syntax in 1.9, we decided to put it under hive dialect. (Subsequent votes will be taken to determine.) MSCK means metastore check, It is listed in the document mainly for the integrity of the story. Of course, it can be ignored. I fully agree that it can be discussed slowly in the future. >The current problem for the CLI is that it still does not use the flink-sql-parser. We can support all new syntax in Flink 1.11 once the CLI architecture has changed there. With the current architecture (using simple regex), I'm not sure if we can support all the grammar listed in the FLIP. You are right, we can not support any DDL grammars in FLIP. we should wait refactor of SQL-CLI in 1.11. What we can do is to remove the restriction of hiveDialect for the insert (overwrite and partition) syntax. That's what we've supported, and so far we haven't received any objections. Best, Jingsong Lee On Fri, Dec 13, 2019 at 4:51 PM Timo Walther wrote: > Hi everyone, > > sorry, I was not aware that FLIP-63 already lists a lot of additional > SQL grammar. It was accepted though an official voting process so I > guess we can adopt the listed grammar for Flink SQL. > > The only thing that confuses me is the mentioning of `The DDL can like > this (With hive dialect)`, for the next time we should make it more > explicit what belongs to the Flink SQL dialect and what belongs to the > Hive dialect. Also the not intuitive `MSCK REPAIR TABLE` seems strange > to me. What does MSCK stand for? Maybe we should skip stuff like that > for now. > > The current problem for the CLI is that it still does not use the > flink-sql-parser. We can support all new syntax in Flink 1.11 once the > CLI architecture has changed there. With the current architecture (using > simple regex), I'm not sure if we can support all the grammar listed in > the FLIP. > > Regards, > Timo > > On 13.12.19 04:22, Rui Li wrote: > > Hi Timo, > > > > I understand we need further discussion about syntax/dialect for 1.11. > But > > as Jark has pointed out, the current implementation violates the accepted > > design of FLIP-63, which IMO qualifies as a bug. Given that it's a bug > and > > has great impact on the usability of our Hive integration, do you think > we > > can fix it in 1.10? > > > > On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li > wrote: > > > >> Hi Timo, > >> > >> I am OK if you think they are not bug and they should not be included in > >> 1.10. > >> > >> I think they have been accepted in FLIP-63. And there is no objection. > It > >> has been more than three months since the discussion of FLIP-63. It's > been > >> six months since Flink added these two syntaxs. > >> > >> But I can also start discussion and vote thread for FLIP-63 again, to > make > >> sure once again that everyone is happy. > >> > >> Best, > >> Jingsong Lee > >> > >> On Thu, Dec 12, 2019 at 11:35 PM Timo Walther > wrote: > >> > >>> Hi Jingsong, > >>> > >>> I will also add my opinion here for future discussions: > >>> > >>> We had long discussions around SQL syntax in the past (e.g. for > >>> WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in > the > >>> end all parties were happy and we came up with a good long-term > solution > >>> that is unlikely to be changed in the near future. IMHO we can differ > >>> from the standard esp. for DDL statements (every SQL vendors has custom > >>> syntax there) but still we should have time to hear many opinions. > >>> However, for DQL and DML statements we need to be even more cautious > >>> because here we enter SQL standard areas. > >>> > >>> Happy to discuss a partion syntax for 1.11 and go with the solution > that > >>> Jark proposed using a config option. > >>> > >>> Thanks, > >>> Timo > >>> > >>> > >>> On 12.12.19 09:40, Jark Wu wrote: > Hi Jingsong, > > Thanks for the explanation, I think I misunderstood your point at the > begining. > As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are > >>> added > to Flink's > SQL syntax, but CREATE PARTITION TABLE should be limited under Hive > dialect. > However, the current implementation is opposite, INSERT OVERWRITE > and INSERT PARTITION are > under the dialect limitation, but CREATE PARTITION TABLE is not. > > So it is indeed a bug which should be fixed in 1.10. > > Best, > Jark > > On Thu, 12 Dec 2019 at 16:35, Jingsong Li > >>> wrote: > > > Hi Jark, > > > > Let's recall FLIP-63, > > We supported these syntax in hive dialect at 1.9. All of my reasons > >> for > > launching FLIP-63 are to bring partition support to Flink itself. > > Not only batch, but also we have the need to stream jobs t
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi everyone, sorry, I was not aware that FLIP-63 already lists a lot of additional SQL grammar. It was accepted though an official voting process so I guess we can adopt the listed grammar for Flink SQL. The only thing that confuses me is the mentioning of `The DDL can like this (With hive dialect)`, for the next time we should make it more explicit what belongs to the Flink SQL dialect and what belongs to the Hive dialect. Also the not intuitive `MSCK REPAIR TABLE` seems strange to me. What does MSCK stand for? Maybe we should skip stuff like that for now. The current problem for the CLI is that it still does not use the flink-sql-parser. We can support all new syntax in Flink 1.11 once the CLI architecture has changed there. With the current architecture (using simple regex), I'm not sure if we can support all the grammar listed in the FLIP. Regards, Timo On 13.12.19 04:22, Rui Li wrote: Hi Timo, I understand we need further discussion about syntax/dialect for 1.11. But as Jark has pointed out, the current implementation violates the accepted design of FLIP-63, which IMO qualifies as a bug. Given that it's a bug and has great impact on the usability of our Hive integration, do you think we can fix it in 1.10? On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li wrote: Hi Timo, I am OK if you think they are not bug and they should not be included in 1.10. I think they have been accepted in FLIP-63. And there is no objection. It has been more than three months since the discussion of FLIP-63. It's been six months since Flink added these two syntaxs. But I can also start discussion and vote thread for FLIP-63 again, to make sure once again that everyone is happy. Best, Jingsong Lee On Thu, Dec 12, 2019 at 11:35 PM Timo Walther wrote: Hi Jingsong, I will also add my opinion here for future discussions: We had long discussions around SQL syntax in the past (e.g. for WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the end all parties were happy and we came up with a good long-term solution that is unlikely to be changed in the near future. IMHO we can differ from the standard esp. for DDL statements (every SQL vendors has custom syntax there) but still we should have time to hear many opinions. However, for DQL and DML statements we need to be even more cautious because here we enter SQL standard areas. Happy to discuss a partion syntax for 1.11 and go with the solution that Jark proposed using a config option. Thanks, Timo On 12.12.19 09:40, Jark Wu wrote: Hi Jingsong, Thanks for the explanation, I think I misunderstood your point at the begining. As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are added to Flink's SQL syntax, but CREATE PARTITION TABLE should be limited under Hive dialect. However, the current implementation is opposite, INSERT OVERWRITE and INSERT PARTITION are under the dialect limitation, but CREATE PARTITION TABLE is not. So it is indeed a bug which should be fixed in 1.10. Best, Jark On Thu, 12 Dec 2019 at 16:35, Jingsong Li wrote: Hi Jark, Let's recall FLIP-63, We supported these syntax in hive dialect at 1.9. All of my reasons for launching FLIP-63 are to bring partition support to Flink itself. Not only batch, but also we have the need to stream jobs to write partition files today, which is also one of our very important application scenarios. The original intention of FLIP-63 is to bring all partition syntax to Flink, but in the end you and I have some different opinion in creating partition table, so our consensus is to leave it in hive dialect, only it. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support Best, Jingsong Lee On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote: Hi jingsong, Watermark is not a standard syntax, that's why we had a FLIP and long discussion to add it to Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and PARTITION syntax to Flink's own syntax, we also need a FLIP or a VOTE, and this may can't happen soon (we should hear more people's opinions on this). Regarding to the sql-dialect configuration, I was not saying to involve the whole FLIP-89. I mean we can just start a VOTE to expose it as `table.planner.sql-dialect` and include it in 1.10. The change can be very small, by adding a ConfigOption and changing the implementation of TableConfig#getSqlDialect/setSqlDialect. I believe it is smaller and safer than changing the parser. Btw, I cc'ed Yu Li and Gary into the discussion, because release managers should be aware of this. Best, Jark On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote: Thanks Jingsong for bring up this discussion ~ After reviewing FLIP-63, it seems that we have made a conclusion for the syntax - INSERT OVERWRITE ... - INSERT INTO … PARTITION Which means that they should not have the Hive dialect limitation, so I’m inclined that the behaviors for SQL-C
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Timo, I understand we need further discussion about syntax/dialect for 1.11. But as Jark has pointed out, the current implementation violates the accepted design of FLIP-63, which IMO qualifies as a bug. Given that it's a bug and has great impact on the usability of our Hive integration, do you think we can fix it in 1.10? On Fri, Dec 13, 2019 at 12:24 AM Jingsong Li wrote: > Hi Timo, > > I am OK if you think they are not bug and they should not be included in > 1.10. > > I think they have been accepted in FLIP-63. And there is no objection. It > has been more than three months since the discussion of FLIP-63. It's been > six months since Flink added these two syntaxs. > > But I can also start discussion and vote thread for FLIP-63 again, to make > sure once again that everyone is happy. > > Best, > Jingsong Lee > > On Thu, Dec 12, 2019 at 11:35 PM Timo Walther wrote: > > > Hi Jingsong, > > > > I will also add my opinion here for future discussions: > > > > We had long discussions around SQL syntax in the past (e.g. for > > WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the > > end all parties were happy and we came up with a good long-term solution > > that is unlikely to be changed in the near future. IMHO we can differ > > from the standard esp. for DDL statements (every SQL vendors has custom > > syntax there) but still we should have time to hear many opinions. > > However, for DQL and DML statements we need to be even more cautious > > because here we enter SQL standard areas. > > > > Happy to discuss a partion syntax for 1.11 and go with the solution that > > Jark proposed using a config option. > > > > Thanks, > > Timo > > > > > > On 12.12.19 09:40, Jark Wu wrote: > > > Hi Jingsong, > > > > > > Thanks for the explanation, I think I misunderstood your point at the > > > begining. > > > As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are > > added > > > to Flink's > > > SQL syntax, but CREATE PARTITION TABLE should be limited under Hive > > > dialect. > > > However, the current implementation is opposite, INSERT OVERWRITE > > > and INSERT PARTITION are > > > under the dialect limitation, but CREATE PARTITION TABLE is not. > > > > > > So it is indeed a bug which should be fixed in 1.10. > > > > > > Best, > > > Jark > > > > > > On Thu, 12 Dec 2019 at 16:35, Jingsong Li > > wrote: > > > > > >> Hi Jark, > > >> > > >> Let's recall FLIP-63, > > >> We supported these syntax in hive dialect at 1.9. All of my reasons > for > > >> launching FLIP-63 are to bring partition support to Flink itself. > > >> Not only batch, but also we have the need to stream jobs to write > > partition > > >> files today, which is also one of our very important application > > scenarios. > > >> > > >> The original intention of FLIP-63 is to bring all partition syntax to > > >> Flink, but in the end you and I have some different opinion in > creating > > >> partition table, so our consensus is to leave it in hive dialect, only > > it. > > >> > > >> [1] > > >> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > >> > > >> Best, > > >> Jingsong Lee > > >> > > >> On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote: > > >> > > >>> Hi jingsong, > > >>> > > >>> Watermark is not a standard syntax, that's why we had a FLIP and long > > >>> discussion to add it to > > >>> Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and > > >>> PARTITION syntax to > > >>> Flink's own syntax, we also need a FLIP or a VOTE, and this may > > can't > > >>> happen soon (we should > > >>> hear more people's opinions on this). > > >>> > > >>> Regarding to the sql-dialect configuration, I was not saying to > involve > > >> the > > >>> whole FLIP-89. I mean we can just > > >>> start a VOTE to expose it as `table.planner.sql-dialect` and include > it > > >> in > > >>> 1.10. The change can be very small, by > > >>> adding a ConfigOption and changing the implementation of > > >>> TableConfig#getSqlDialect/setSqlDialect. I believe > > >>> it is smaller and safer than changing the parser. > > >>> > > >>> Btw, I cc'ed Yu Li and Gary into the discussion, because release > > managers > > >>> should be aware of this. > > >>> > > >>> Best, > > >>> Jark > > >>> > > >>> > > >>> > > >>> On Thu, 12 Dec 2019 at 11:47, Danny Chan > wrote: > > >>> > > Thanks Jingsong for bring up this discussion ~ > > > > After reviewing FLIP-63, it seems that we have made a conclusion for > > >> the > > syntax > > > > - INSERT OVERWRITE ... > > - INSERT INTO … PARTITION > > > > Which means that they should not have the Hive dialect limitation, > so > > >> I’m > > inclined that the behaviors for SQL-CLI is unexpected, or a “bug” > that > > >>> need > > to fix. > > > > We did not make a conclusion for the syntax: > > > > - CREATE TABLE … PARTITIONED BY ... > > > > Which means that the behavior of i
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Timo, I am OK if you think they are not bug and they should not be included in 1.10. I think they have been accepted in FLIP-63. And there is no objection. It has been more than three months since the discussion of FLIP-63. It's been six months since Flink added these two syntaxs. But I can also start discussion and vote thread for FLIP-63 again, to make sure once again that everyone is happy. Best, Jingsong Lee On Thu, Dec 12, 2019 at 11:35 PM Timo Walther wrote: > Hi Jingsong, > > I will also add my opinion here for future discussions: > > We had long discussions around SQL syntax in the past (e.g. for > WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the > end all parties were happy and we came up with a good long-term solution > that is unlikely to be changed in the near future. IMHO we can differ > from the standard esp. for DDL statements (every SQL vendors has custom > syntax there) but still we should have time to hear many opinions. > However, for DQL and DML statements we need to be even more cautious > because here we enter SQL standard areas. > > Happy to discuss a partion syntax for 1.11 and go with the solution that > Jark proposed using a config option. > > Thanks, > Timo > > > On 12.12.19 09:40, Jark Wu wrote: > > Hi Jingsong, > > > > Thanks for the explanation, I think I misunderstood your point at the > > begining. > > As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are > added > > to Flink's > > SQL syntax, but CREATE PARTITION TABLE should be limited under Hive > > dialect. > > However, the current implementation is opposite, INSERT OVERWRITE > > and INSERT PARTITION are > > under the dialect limitation, but CREATE PARTITION TABLE is not. > > > > So it is indeed a bug which should be fixed in 1.10. > > > > Best, > > Jark > > > > On Thu, 12 Dec 2019 at 16:35, Jingsong Li > wrote: > > > >> Hi Jark, > >> > >> Let's recall FLIP-63, > >> We supported these syntax in hive dialect at 1.9. All of my reasons for > >> launching FLIP-63 are to bring partition support to Flink itself. > >> Not only batch, but also we have the need to stream jobs to write > partition > >> files today, which is also one of our very important application > scenarios. > >> > >> The original intention of FLIP-63 is to bring all partition syntax to > >> Flink, but in the end you and I have some different opinion in creating > >> partition table, so our consensus is to leave it in hive dialect, only > it. > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >> > >> Best, > >> Jingsong Lee > >> > >> On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote: > >> > >>> Hi jingsong, > >>> > >>> Watermark is not a standard syntax, that's why we had a FLIP and long > >>> discussion to add it to > >>> Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and > >>> PARTITION syntax to > >>> Flink's own syntax, we also need a FLIP or a VOTE, and this may > can't > >>> happen soon (we should > >>> hear more people's opinions on this). > >>> > >>> Regarding to the sql-dialect configuration, I was not saying to involve > >> the > >>> whole FLIP-89. I mean we can just > >>> start a VOTE to expose it as `table.planner.sql-dialect` and include it > >> in > >>> 1.10. The change can be very small, by > >>> adding a ConfigOption and changing the implementation of > >>> TableConfig#getSqlDialect/setSqlDialect. I believe > >>> it is smaller and safer than changing the parser. > >>> > >>> Btw, I cc'ed Yu Li and Gary into the discussion, because release > managers > >>> should be aware of this. > >>> > >>> Best, > >>> Jark > >>> > >>> > >>> > >>> On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote: > >>> > Thanks Jingsong for bring up this discussion ~ > > After reviewing FLIP-63, it seems that we have made a conclusion for > >> the > syntax > > - INSERT OVERWRITE ... > - INSERT INTO … PARTITION > > Which means that they should not have the Hive dialect limitation, so > >> I’m > inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that > >>> need > to fix. > > We did not make a conclusion for the syntax: > > - CREATE TABLE … PARTITIONED BY ... > > Which means that the behavior of it is under-discussion, so it is okey > >> to > be without the HIVE dialect limitation, we do not actually have any > >> table > sources/sinks that support such a DDL so for current code base, users > should not be influenced by the behaviors change. > > So I’m > > +1 to remove the hive dialect limitations for INSERT OVERWRITE and > >> INSERT > PARTITION > +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished > yet, we better do this until FLIP-89 is resolved. > > Best, > Danny Chan > 在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道： > > Hi Dev, > > > > After cutting
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Jingsong,
I will also add my opinion here for future discussions:
We had long discussions around SQL syntax in the past (e.g. for
WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the
end all parties were happy and we came up with a good long-term solution
that is unlikely to be changed in the near future. IMHO we can differ
from the standard esp. for DDL statements (every SQL vendors has custom
syntax there) but still we should have time to hear many opinions.
However, for DQL and DML statements we need to be even more cautious
because here we enter SQL standard areas.
Happy to discuss a partion syntax for 1.11 and go with the solution that
Jark proposed using a config option.
Thanks,
Timo
On 12.12.19 09:40, Jark Wu wrote:
Hi Jingsong,
Thanks for the explanation, I think I misunderstood your point at the
begining.
As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are added
to Flink's
SQL syntax, but CREATE PARTITION TABLE should be limited under Hive
dialect.
However, the current implementation is opposite, INSERT OVERWRITE
and INSERT PARTITION are
under the dialect limitation, but CREATE PARTITION TABLE is not.
So it is indeed a bug which should be fixed in 1.10.
Best,
Jark
On Thu, 12 Dec 2019 at 16:35, Jingsong Li wrote:
Hi Jark,
Let's recall FLIP-63,
We supported these syntax in hive dialect at 1.9. All of my reasons for
launching FLIP-63 are to bring partition support to Flink itself.
Not only batch, but also we have the need to stream jobs to write partition
files today, which is also one of our very important application scenarios.
The original intention of FLIP-63 is to bring all partition syntax to
Flink, but in the end you and I have some different opinion in creating
partition table, so our consensus is to leave it in hive dialect, only it.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
Best,
Jingsong Lee
On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote:
Hi jingsong,
Watermark is not a standard syntax, that's why we had a FLIP and long
discussion to add it to
Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and
PARTITION syntax to
Flink's own syntax, we also need a FLIP or a VOTE, and this may can't
happen soon (we should
hear more people's opinions on this).
Regarding to the sql-dialect configuration, I was not saying to involve
the
whole FLIP-89. I mean we can just
start a VOTE to expose it as `table.planner.sql-dialect` and include it
in
1.10. The change can be very small, by
adding a ConfigOption and changing the implementation of
TableConfig#getSqlDialect/setSqlDialect. I believe
it is smaller and safer than changing the parser.
Btw, I cc'ed Yu Li and Gary into the discussion, because release managers
should be aware of this.
Best,
Jark
On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote:
Thanks Jingsong for bring up this discussion ~
After reviewing FLIP-63, it seems that we have made a conclusion for
the
syntax
- INSERT OVERWRITE ...
- INSERT INTO … PARTITION
Which means that they should not have the Hive dialect limitation, so
I’m
inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that
need
to fix.
We did not make a conclusion for the syntax:
- CREATE TABLE … PARTITIONED BY ...
Which means that the behavior of it is under-discussion, so it is okey
to
be without the HIVE dialect limitation, we do not actually have any
table
sources/sinks that support such a DDL so for current code base, users
should not be influenced by the behaviors change.
So I’m
+1 to remove the hive dialect limitations for INSERT OVERWRITE and
INSERT
PARTITION
+0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished
yet, we better do this until FLIP-89 is resolved.
Best,
Danny Chan
在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道：
Hi Dev,
After cutting out the branch of 1.10, I tried the following functions
of
SQL-CLI and found that it does not support:
- insert overwrite
- PARTITION (partcol1=val1, partcol2=val2 ...)
The SQL pattern is:
INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION
(partcol1=val1,
partcol2=val2 ...) select_statement1 FROM from_statement;
It is a surprise to me.
The reason is that we only allow these two grammars in hive dialect.
And
SQL-CLI does not have an interface to switch dialects.
Because it directly hinders the SQL-CLI's insert syntax in hive
integration
and seriously hinders the practicability of SQL-CLI.
And we have introduced these two grammars in FLIP-63 [1] to Flink.
Here are my question:
1.Should we remove hive dialect limitation for these two grammars?
2.Should we fix this in 1.10?
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
Best,
Jingsong Lee
--
Best, Jingsong Lee
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Jingsong,
Thanks for the explanation, I think I misunderstood your point at the
begining.
As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are added
to Flink's
SQL syntax, but CREATE PARTITION TABLE should be limited under Hive
dialect.
However, the current implementation is opposite, INSERT OVERWRITE
and INSERT PARTITION are
under the dialect limitation, but CREATE PARTITION TABLE is not.
So it is indeed a bug which should be fixed in 1.10.
Best,
Jark
On Thu, 12 Dec 2019 at 16:35, Jingsong Li wrote:
> Hi Jark,
>
> Let's recall FLIP-63,
> We supported these syntax in hive dialect at 1.9. All of my reasons for
> launching FLIP-63 are to bring partition support to Flink itself.
> Not only batch, but also we have the need to stream jobs to write partition
> files today, which is also one of our very important application scenarios.
>
> The original intention of FLIP-63 is to bring all partition syntax to
> Flink, but in the end you and I have some different opinion in creating
> partition table, so our consensus is to leave it in hive dialect, only it.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>
> Best,
> Jingsong Lee
>
> On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote:
>
> > Hi jingsong,
> >
> > Watermark is not a standard syntax, that's why we had a FLIP and long
> > discussion to add it to
> > Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and
> > PARTITION syntax to
> > Flink's own syntax, we also need a FLIP or a VOTE, and this may can't
> > happen soon (we should
> > hear more people's opinions on this).
> >
> > Regarding to the sql-dialect configuration, I was not saying to involve
> the
> > whole FLIP-89. I mean we can just
> > start a VOTE to expose it as `table.planner.sql-dialect` and include it
> in
> > 1.10. The change can be very small, by
> > adding a ConfigOption and changing the implementation of
> > TableConfig#getSqlDialect/setSqlDialect. I believe
> > it is smaller and safer than changing the parser.
> >
> > Btw, I cc'ed Yu Li and Gary into the discussion, because release managers
> > should be aware of this.
> >
> > Best,
> > Jark
> >
> >
> >
> > On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote:
> >
> > > Thanks Jingsong for bring up this discussion ~
> > >
> > > After reviewing FLIP-63, it seems that we have made a conclusion for
> the
> > > syntax
> > >
> > > - INSERT OVERWRITE ...
> > > - INSERT INTO … PARTITION
> > >
> > > Which means that they should not have the Hive dialect limitation, so
> I’m
> > > inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that
> > need
> > > to fix.
> > >
> > > We did not make a conclusion for the syntax:
> > >
> > > - CREATE TABLE … PARTITIONED BY ...
> > >
> > > Which means that the behavior of it is under-discussion, so it is okey
> to
> > > be without the HIVE dialect limitation, we do not actually have any
> table
> > > sources/sinks that support such a DDL so for current code base, users
> > > should not be influenced by the behaviors change.
> > >
> > > So I’m
> > >
> > > +1 to remove the hive dialect limitations for INSERT OVERWRITE and
> INSERT
> > > PARTITION
> > > +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished
> > > yet, we better do this until FLIP-89 is resolved.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道：
> > > > Hi Dev,
> > > >
> > > > After cutting out the branch of 1.10, I tried the following functions
> > of
> > > > SQL-CLI and found that it does not support:
> > > > - insert overwrite
> > > > - PARTITION (partcol1=val1, partcol2=val2 ...)
> > > > The SQL pattern is:
> > > > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION
> (partcol1=val1,
> > > > partcol2=val2 ...) select_statement1 FROM from_statement;
> > > > It is a surprise to me.
> > > > The reason is that we only allow these two grammars in hive dialect.
> > And
> > > > SQL-CLI does not have an interface to switch dialects.
> > > >
> > > > Because it directly hinders the SQL-CLI's insert syntax in hive
> > > integration
> > > > and seriously hinders the practicability of SQL-CLI.
> > > > And we have introduced these two grammars in FLIP-63 [1] to Flink.
> > > > Here are my question:
> > > > 1.Should we remove hive dialect limitation for these two grammars?
> > > > 2.Should we fix this in 1.10?
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> > > >
> > > > Best,
> > > > Jingsong Lee
> > >
> >
>
>
> --
> Best, Jingsong Lee
>
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Jark,
Let's recall FLIP-63,
We supported these syntax in hive dialect at 1.9. All of my reasons for
launching FLIP-63 are to bring partition support to Flink itself.
Not only batch, but also we have the need to stream jobs to write partition
files today, which is also one of our very important application scenarios.
The original intention of FLIP-63 is to bring all partition syntax to
Flink, but in the end you and I have some different opinion in creating
partition table, so our consensus is to leave it in hive dialect, only it.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
Best,
Jingsong Lee
On Thu, Dec 12, 2019 at 4:05 PM Jark Wu wrote:
> Hi jingsong,
>
> Watermark is not a standard syntax, that's why we had a FLIP and long
> discussion to add it to
> Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and
> PARTITION syntax to
> Flink's own syntax, we also need a FLIP or a VOTE, and this may can't
> happen soon (we should
> hear more people's opinions on this).
>
> Regarding to the sql-dialect configuration, I was not saying to involve the
> whole FLIP-89. I mean we can just
> start a VOTE to expose it as `table.planner.sql-dialect` and include it in
> 1.10. The change can be very small, by
> adding a ConfigOption and changing the implementation of
> TableConfig#getSqlDialect/setSqlDialect. I believe
> it is smaller and safer than changing the parser.
>
> Btw, I cc'ed Yu Li and Gary into the discussion, because release managers
> should be aware of this.
>
> Best,
> Jark
>
>
>
> On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote:
>
> > Thanks Jingsong for bring up this discussion ~
> >
> > After reviewing FLIP-63, it seems that we have made a conclusion for the
> > syntax
> >
> > - INSERT OVERWRITE ...
> > - INSERT INTO … PARTITION
> >
> > Which means that they should not have the Hive dialect limitation, so I’m
> > inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that
> need
> > to fix.
> >
> > We did not make a conclusion for the syntax:
> >
> > - CREATE TABLE … PARTITIONED BY ...
> >
> > Which means that the behavior of it is under-discussion, so it is okey to
> > be without the HIVE dialect limitation, we do not actually have any table
> > sources/sinks that support such a DDL so for current code base, users
> > should not be influenced by the behaviors change.
> >
> > So I’m
> >
> > +1 to remove the hive dialect limitations for INSERT OVERWRITE and INSERT
> > PARTITION
> > +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished
> > yet, we better do this until FLIP-89 is resolved.
> >
> > Best,
> > Danny Chan
> > 在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道：
> > > Hi Dev,
> > >
> > > After cutting out the branch of 1.10, I tried the following functions
> of
> > > SQL-CLI and found that it does not support:
> > > - insert overwrite
> > > - PARTITION (partcol1=val1, partcol2=val2 ...)
> > > The SQL pattern is:
> > > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> > > partcol2=val2 ...) select_statement1 FROM from_statement;
> > > It is a surprise to me.
> > > The reason is that we only allow these two grammars in hive dialect.
> And
> > > SQL-CLI does not have an interface to switch dialects.
> > >
> > > Because it directly hinders the SQL-CLI's insert syntax in hive
> > integration
> > > and seriously hinders the practicability of SQL-CLI.
> > > And we have introduced these two grammars in FLIP-63 [1] to Flink.
> > > Here are my question:
> > > 1.Should we remove hive dialect limitation for these two grammars?
> > > 2.Should we fix this in 1.10?
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> > >
> > > Best,
> > > Jingsong Lee
> >
>
--
Best, Jingsong Lee
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi jingsong,
Watermark is not a standard syntax, that's why we had a FLIP and long
discussion to add it to
Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and
PARTITION syntax to
Flink's own syntax, we also need a FLIP or a VOTE, and this may can't
happen soon (we should
hear more people's opinions on this).
Regarding to the sql-dialect configuration, I was not saying to involve the
whole FLIP-89. I mean we can just
start a VOTE to expose it as `table.planner.sql-dialect` and include it in
1.10. The change can be very small, by
adding a ConfigOption and changing the implementation of
TableConfig#getSqlDialect/setSqlDialect. I believe
it is smaller and safer than changing the parser.
Btw, I cc'ed Yu Li and Gary into the discussion, because release managers
should be aware of this.
Best,
Jark
On Thu, 12 Dec 2019 at 11:47, Danny Chan wrote:
> Thanks Jingsong for bring up this discussion ~
>
> After reviewing FLIP-63, it seems that we have made a conclusion for the
> syntax
>
> - INSERT OVERWRITE ...
> - INSERT INTO … PARTITION
>
> Which means that they should not have the Hive dialect limitation, so I’m
> inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that need
> to fix.
>
> We did not make a conclusion for the syntax:
>
> - CREATE TABLE … PARTITIONED BY ...
>
> Which means that the behavior of it is under-discussion, so it is okey to
> be without the HIVE dialect limitation, we do not actually have any table
> sources/sinks that support such a DDL so for current code base, users
> should not be influenced by the behaviors change.
>
> So I’m
>
> +1 to remove the hive dialect limitations for INSERT OVERWRITE and INSERT
> PARTITION
> +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished
> yet, we better do this until FLIP-89 is resolved.
>
> Best,
> Danny Chan
> 在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道：
> > Hi Dev,
> >
> > After cutting out the branch of 1.10, I tried the following functions of
> > SQL-CLI and found that it does not support:
> > - insert overwrite
> > - PARTITION (partcol1=val1, partcol2=val2 ...)
> > The SQL pattern is:
> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> > partcol2=val2 ...) select_statement1 FROM from_statement;
> > It is a surprise to me.
> > The reason is that we only allow these two grammars in hive dialect. And
> > SQL-CLI does not have an interface to switch dialects.
> >
> > Because it directly hinders the SQL-CLI's insert syntax in hive
> integration
> > and seriously hinders the practicability of SQL-CLI.
> > And we have introduced these two grammars in FLIP-63 [1] to Flink.
> > Here are my question:
> > 1.Should we remove hive dialect limitation for these two grammars?
> > 2.Should we fix this in 1.10?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> >
> > Best,
> > Jingsong Lee
>
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Thanks Jingsong for bring up this discussion ~
After reviewing FLIP-63, it seems that we have made a conclusion for the syntax
- INSERT OVERWRITE ...
- INSERT INTO … PARTITION
Which means that they should not have the Hive dialect limitation, so I’m
inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that need to
fix.
We did not make a conclusion for the syntax:
- CREATE TABLE … PARTITIONED BY ...
Which means that the behavior of it is under-discussion, so it is okey to be
without the HIVE dialect limitation, we do not actually have any table
sources/sinks that support such a DDL so for current code base, users should
not be influenced by the behaviors change.
So I’m
+1 to remove the hive dialect limitations for INSERT OVERWRITE and INSERT
PARTITION
+0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished yet, we
better do this until FLIP-89 is resolved.
Best,
Danny Chan
在 2019年12月11日 +0800 PM5:29，Jingsong Li ，写道：
> Hi Dev,
>
> After cutting out the branch of 1.10, I tried the following functions of
> SQL-CLI and found that it does not support:
> - insert overwrite
> - PARTITION (partcol1=val1, partcol2=val2 ...)
> The SQL pattern is:
> INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> partcol2=val2 ...) select_statement1 FROM from_statement;
> It is a surprise to me.
> The reason is that we only allow these two grammars in hive dialect. And
> SQL-CLI does not have an interface to switch dialects.
>
> Because it directly hinders the SQL-CLI's insert syntax in hive integration
> and seriously hinders the practicability of SQL-CLI.
> And we have introduced these two grammars in FLIP-63 [1] to Flink.
> Here are my question:
> 1.Should we remove hive dialect limitation for these two grammars?
> 2.Should we fix this in 1.10?
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>
> Best,
> Jingsong Lee
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Jark, > The dialect restriction is introduced on purpose, because OVERWRITE and PARTITION syntax are not SQL standard. My understanding is that watermark [1] is also a non-standard grammar. We can extend SQL standard syntax. > Even in the discussion of FLIP-63, the community have different opinion on whether the partition fields should be declared in schema part or not [1]. The status of FLIP-63 is that limit the proposed syntax in Hive dialect, and we may propose new built-in syntax for creating partition tables in the future. No, I'm not talking about creating partition tables. I'm talking about inserting tables. FLIP-63 doesn't mean it's only for hive dialect. Flink itself can need the support of partition table. I'm sure it will further strengthen the work in 1.11. I remember that we only had disputes on creating partition tables, so I moved it to the chapter to be discussed, and limited it to hive dialect, only creating partition tables, right? > FLIP-89 proposed to make TableConfig configurable including the dialect configuration, i.g. `table.planner.sql-dialect`. So the only thing we need to do is introducing such a configuration. I'm not sure we should introduce FLIP-89, and dialect has no other function except for overwrite and partition inserting at present, I think they should be Flink syntax. And remove hive dialect limitation is a very small change/fix, Just remove a few lines of check. What do you think? (If you think so, after some discussion, we can start a vote later.) [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL Best, Jingsong Lee On Thu, Dec 12, 2019 at 11:11 AM Rui Li wrote: > +1 to fix it in 1.10. If this feature doesn't work via SQL CLI, I guess it > doesn't work for most Hive users. > +0 to remove the dialect check. I don't see much benefit this check can > bring to users, except that it prevents users from accidentally using some > Hive features, which doesn't seem to be very harmful anyway. But I don't > have a strong opinion for it and it probably needs more thorough > discussion. > > > On Thu, Dec 12, 2019 at 7:49 AM Jark Wu wrote: > > > Thanks Jingsong, > > > > OVERWRITE and PARTITION are very fundamental features for Hive users. > > I'm sorry to hear that it doesn't work in SQL CLI. > > > > > Remove hive dialect limitation for these two grammars? > > The dialect restriction is introduced on purpose, because OVERWRITE and > > PARTITION syntax > > are not SQL standard. Even in the discussion of FLIP-63, the community > > have different opinion on > > whether the partition fields should be declared in schema part or not > [1]. > > The status of FLIP-63 is > > that limit the proposed syntax in Hive dialect, and we may propose new > > built-in syntax for creating > > partition tables in the future. So I'm -1 to removing dialect > limitation. > > > > > Should we fix this in 1.10? > > From my point of view, the problem is that users can't switch dialects in > > SQL CLI, because it is not > > exposed as a configuration. FLIP-89 [2] proposed to make TableConfig > > configurable including the > > dialect configuration, i.g. `table.planner.sql-dialect`. So the only > > thing we need to do is introducing > > such a configuration, but this is definitely is not a *bug*. However, > > considering it is a small change, > > and from user perspective, I'm +1 to introducing such a configuration. > > > > Best, > > Jark > > > > [1]: > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html > > [2]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig > > > > On Thu, 12 Dec 2019 at 03:24, Bowen Li wrote: > > > >> Hi Jingsong, > >> > >> Thanks a lot for reporting this issue. > >> > >> IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support > Hive > >> integration before FLIP-63 was proposed to introduce generic partition > >> support to Flink. Thus when we added these syntax, we were intentionally > >> conservative and limited their scope to Hive dialect. @Rui may help to > >> confirm that. I'm a bit surprised about it as well. As core APIs of > >> FLIP-63 > >> were done, I don't see why we would limit these syntax to Hive dialect > >> alone. It's just unfortunately that we may have forgot to revisit this > >> topic and we apparently missed some test cases on SQL CLI side. Sorry > for > >> that. > >> > >> From a product perspective, SQL CLI is super critical for Flink-Hive > >> integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are > two > >> of the most commonly used syntax in Hive, the product wouldn't be > useable > >> without such proper support. > >> > >> Thus, I think this is a *bug*, we can stop limiting them in Hive > dialect, > >> and should fix it in 1.10. We should also add more test coverage for SQL > >> CLI to avoid such surprise. > >> > >> Cheers, > >> Bowen > >> >
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
+1 to fix it in 1.10. If this feature doesn't work via SQL CLI, I guess it
doesn't work for most Hive users.
+0 to remove the dialect check. I don't see much benefit this check can
bring to users, except that it prevents users from accidentally using some
Hive features, which doesn't seem to be very harmful anyway. But I don't
have a strong opinion for it and it probably needs more thorough discussion.
On Thu, Dec 12, 2019 at 7:49 AM Jark Wu wrote:
> Thanks Jingsong,
>
> OVERWRITE and PARTITION are very fundamental features for Hive users.
> I'm sorry to hear that it doesn't work in SQL CLI.
>
> > Remove hive dialect limitation for these two grammars?
> The dialect restriction is introduced on purpose, because OVERWRITE and
> PARTITION syntax
> are not SQL standard. Even in the discussion of FLIP-63, the community
> have different opinion on
> whether the partition fields should be declared in schema part or not [1].
> The status of FLIP-63 is
> that limit the proposed syntax in Hive dialect, and we may propose new
> built-in syntax for creating
> partition tables in the future. So I'm -1 to removing dialect limitation.
>
> > Should we fix this in 1.10?
> From my point of view, the problem is that users can't switch dialects in
> SQL CLI, because it is not
> exposed as a configuration. FLIP-89 [2] proposed to make TableConfig
> configurable including the
> dialect configuration, i.g. `table.planner.sql-dialect`. So the only
> thing we need to do is introducing
> such a configuration, but this is definitely is not a *bug*. However,
> considering it is a small change,
> and from user perspective, I'm +1 to introducing such a configuration.
>
> Best,
> Jark
>
> [1]:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html
> [2]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig
>
> On Thu, 12 Dec 2019 at 03:24, Bowen Li wrote:
>
>> Hi Jingsong,
>>
>> Thanks a lot for reporting this issue.
>>
>> IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support Hive
>> integration before FLIP-63 was proposed to introduce generic partition
>> support to Flink. Thus when we added these syntax, we were intentionally
>> conservative and limited their scope to Hive dialect. @Rui may help to
>> confirm that. I'm a bit surprised about it as well. As core APIs of
>> FLIP-63
>> were done, I don't see why we would limit these syntax to Hive dialect
>> alone. It's just unfortunately that we may have forgot to revisit this
>> topic and we apparently missed some test cases on SQL CLI side. Sorry for
>> that.
>>
>> From a product perspective, SQL CLI is super critical for Flink-Hive
>> integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are two
>> of the most commonly used syntax in Hive, the product wouldn't be useable
>> without such proper support.
>>
>> Thus, I think this is a *bug*, we can stop limiting them in Hive dialect,
>> and should fix it in 1.10. We should also add more test coverage for SQL
>> CLI to avoid such surprise.
>>
>> Cheers,
>> Bowen
>>
>> On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li
>> wrote:
>>
>> > Hi Dev,
>> >
>> > After cutting out the branch of 1.10, I tried the following functions of
>> > SQL-CLI and found that it does not support:
>> > - insert overwrite
>> > - PARTITION (partcol1=val1, partcol2=val2 ...)
>> > The SQL pattern is:
>> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
>> > partcol2=val2 ...) select_statement1 FROM from_statement;
>> > It is a surprise to me.
>> > The reason is that we only allow these two grammars in hive dialect. And
>> > SQL-CLI does not have an interface to switch dialects.
>> >
>> > Because it directly hinders the SQL-CLI's insert syntax in hive
>> integration
>> > and seriously hinders the practicability of SQL-CLI.
>> > And we have introduced these two grammars in FLIP-63 [1] to Flink.
>> > Here are my question:
>> > 1.Should we remove hive dialect limitation for these two grammars?
>> > 2.Should we fix this in 1.10?
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>> >
>> > Best,
>> > Jingsong Lee
>> >
>>
>
--
Best regards!
Rui Li
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Thanks Jingsong,
OVERWRITE and PARTITION are very fundamental features for Hive users.
I'm sorry to hear that it doesn't work in SQL CLI.
> Remove hive dialect limitation for these two grammars?
The dialect restriction is introduced on purpose, because OVERWRITE and
PARTITION syntax
are not SQL standard. Even in the discussion of FLIP-63, the community have
different opinion on
whether the partition fields should be declared in schema part or not [1].
The status of FLIP-63 is
that limit the proposed syntax in Hive dialect, and we may propose new
built-in syntax for creating
partition tables in the future. So I'm -1 to removing dialect limitation.
> Should we fix this in 1.10?
>From my point of view, the problem is that users can't switch dialects in
SQL CLI, because it is not
exposed as a configuration. FLIP-89 [2] proposed to make TableConfig
configurable including the
dialect configuration, i.g. `table.planner.sql-dialect`. So the only thing
we need to do is introducing
such a configuration, but this is definitely is not a *bug*. However,
considering it is a small change,
and from user perspective, I'm +1 to introducing such a configuration.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html
[2]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig
On Thu, 12 Dec 2019 at 03:24, Bowen Li wrote:
> Hi Jingsong,
>
> Thanks a lot for reporting this issue.
>
> IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support Hive
> integration before FLIP-63 was proposed to introduce generic partition
> support to Flink. Thus when we added these syntax, we were intentionally
> conservative and limited their scope to Hive dialect. @Rui may help to
> confirm that. I'm a bit surprised about it as well. As core APIs of FLIP-63
> were done, I don't see why we would limit these syntax to Hive dialect
> alone. It's just unfortunately that we may have forgot to revisit this
> topic and we apparently missed some test cases on SQL CLI side. Sorry for
> that.
>
> From a product perspective, SQL CLI is super critical for Flink-Hive
> integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are two
> of the most commonly used syntax in Hive, the product wouldn't be useable
> without such proper support.
>
> Thus, I think this is a *bug*, we can stop limiting them in Hive dialect,
> and should fix it in 1.10. We should also add more test coverage for SQL
> CLI to avoid such surprise.
>
> Cheers,
> Bowen
>
> On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li
> wrote:
>
> > Hi Dev,
> >
> > After cutting out the branch of 1.10, I tried the following functions of
> > SQL-CLI and found that it does not support:
> > - insert overwrite
> > - PARTITION (partcol1=val1, partcol2=val2 ...)
> > The SQL pattern is:
> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> > partcol2=val2 ...) select_statement1 FROM from_statement;
> > It is a surprise to me.
> > The reason is that we only allow these two grammars in hive dialect. And
> > SQL-CLI does not have an interface to switch dialects.
> >
> > Because it directly hinders the SQL-CLI's insert syntax in hive
> integration
> > and seriously hinders the practicability of SQL-CLI.
> > And we have introduced these two grammars in FLIP-63 [1] to Flink.
> > Here are my question:
> > 1.Should we remove hive dialect limitation for these two grammars?
> > 2.Should we fix this in 1.10?
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> >
> > Best,
> > Jingsong Lee
> >
>
Re: [DISCUSS] Overwrite and partition inserting support in 1.10
Hi Jingsong,
Thanks a lot for reporting this issue.
IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support Hive
integration before FLIP-63 was proposed to introduce generic partition
support to Flink. Thus when we added these syntax, we were intentionally
conservative and limited their scope to Hive dialect. @Rui may help to
confirm that. I'm a bit surprised about it as well. As core APIs of FLIP-63
were done, I don't see why we would limit these syntax to Hive dialect
alone. It's just unfortunately that we may have forgot to revisit this
topic and we apparently missed some test cases on SQL CLI side. Sorry for
that.
>From a product perspective, SQL CLI is super critical for Flink-Hive
integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are two
of the most commonly used syntax in Hive, the product wouldn't be useable
without such proper support.
Thus, I think this is a *bug*, we can stop limiting them in Hive dialect,
and should fix it in 1.10. We should also add more test coverage for SQL
CLI to avoid such surprise.
Cheers,
Bowen
On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li wrote:
> Hi Dev,
>
> After cutting out the branch of 1.10, I tried the following functions of
> SQL-CLI and found that it does not support:
> - insert overwrite
> - PARTITION (partcol1=val1, partcol2=val2 ...)
> The SQL pattern is:
> INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> partcol2=val2 ...) select_statement1 FROM from_statement;
> It is a surprise to me.
> The reason is that we only allow these two grammars in hive dialect. And
> SQL-CLI does not have an interface to switch dialects.
>
> Because it directly hinders the SQL-CLI's insert syntax in hive integration
> and seriously hinders the practicability of SQL-CLI.
> And we have introduced these two grammars in FLIP-63 [1] to Flink.
> Here are my question:
> 1.Should we remove hive dialect limitation for these two grammars?
> 2.Should we fix this in 1.10?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>
> Best,
> Jingsong Lee
>
[DISCUSS] Overwrite and partition inserting support in 1.10
Hi Dev,
After cutting out the branch of 1.10, I tried the following functions of
SQL-CLI and found that it does not support:
- insert overwrite
- PARTITION (partcol1=val1, partcol2=val2 ...)
The SQL pattern is:
INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
partcol2=val2 ...) select_statement1 FROM from_statement;
It is a surprise to me.
The reason is that we only allow these two grammars in hive dialect. And
SQL-CLI does not have an interface to switch dialects.
Because it directly hinders the SQL-CLI's insert syntax in hive integration
and seriously hinders the practicability of SQL-CLI.
And we have introduced these two grammars in FLIP-63 [1] to Flink.
Here are my question:
1.Should we remove hive dialect limitation for these two grammars?
2.Should we fix this in 1.10?
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
Best,
Jingsong Lee
