[DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi guys. When I recently used flink sql to manage internal metadata (catalog/databases/table/functions), I found that many current flink sql statements do not support filtering or some advanced syntax, however these abilities are very useful to end-users. These are some statements I have collected so far, which are supported on other big data engines, such as spark, hive or presto. I wonder if we can support these abilities? In addition, the subject of this email is named 'Auxiliary Statements' mainly because the alignment of these statements will not have much impact on the core SQL runtime. Support or Not With Advanced Syntax Or Not (in/from or like) show create table Yes Yes show tables Yes Yes show columns Yes Yes show catalogs Yes without filter show databases Yes without filter show functions Yes without filter show views Yes without filter show modules Yes without filter show jars Yes without filter show jobs Yes without filter We can see current flink many sql statements only support showing with full datas, without 'FROM/IN' or 'LIKE' filter clause. Support or Not describe database No describe table Yes describe function No describe query No current flink only supports describing tables. Also, please let me know if there is a mistake. Looking forward to your reply. Best Regards, Ran Tao https://github.com/chucheng92
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi Ran, I think it’s always nice to support new syntax if it’s useful for the users. From my side, your syntax table is broken. Could you share it with a Google doc or create a JIRA issue? Best, Jark > 2023年2月21日 17:51，Ran Tao 写道： > > Hi guys. When I recently used flink sql to manage internal metadata > (catalog/databases/table/functions), > I found that many current flink sql statements do not support filtering or > some advanced syntax, however these abilities are very useful to end-users. > > These are some statements I have collected so far, which are supported on > other big data engines, such as spark, hive or presto. I wonder if we can > support these abilities? > > In addition, the subject of this email is named 'Auxiliary Statements' > mainly because the alignment of these statements will not have much impact > on the core SQL runtime. > > Support or Not > > With Advanced Syntax Or Not (in/from or like) > > show create table > > Yes > > Yes > > show tables > > Yes > > Yes > > show columns > > Yes > > Yes > > show catalogs > > Yes > > without filter > > show databases > > Yes > > without filter > > show functions > > Yes > > without filter > > show views > > Yes > > without filter > > show modules > > Yes > > without filter > > show jars > > Yes > > without filter > > show jobs > > Yes > > without filter > > We can see current flink many sql statements only support showing with full > datas, without 'FROM/IN' or 'LIKE' filter clause. > > Support or Not > > describe database > > No > > describe table > > Yes > > describe function > > No > > describe query > > No > > current flink only supports describing tables. > > Also, please let me know if there is a mistake. Looking forward to your > reply. > > > Best Regards, > Ran Tao > https://github.com/chucheng92
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi, Jark. thanks. I have added a google doc. https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing Jark Wu 于2023年2月21日周二 19:27写道： > Hi Ran, > > I think it’s always nice to support new syntax if it’s useful for the > users. > From my side, your syntax table is broken. Could you share it with a > Google doc or create a JIRA issue? > > Best, > Jark > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > Hi guys. When I recently used flink sql to manage internal metadata > > (catalog/databases/table/functions), > > I found that many current flink sql statements do not support filtering > or > > some advanced syntax, however these abilities are very useful to > end-users. > > > > These are some statements I have collected so far, which are supported on > > other big data engines, such as spark, hive or presto. I wonder if we can > > support these abilities? > > > > In addition, the subject of this email is named 'Auxiliary Statements' > > mainly because the alignment of these statements will not have much > impact > > on the core SQL runtime. > > > > Support or Not > > > > With Advanced Syntax Or Not (in/from or like) > > > > show create table > > > > Yes > > > > Yes > > > > show tables > > > > Yes > > > > Yes > > > > show columns > > > > Yes > > > > Yes > > > > show catalogs > > > > Yes > > > > without filter > > > > show databases > > > > Yes > > > > without filter > > > > show functions > > > > Yes > > > > without filter > > > > show views > > > > Yes > > > > without filter > > > > show modules > > > > Yes > > > > without filter > > > > show jars > > > > Yes > > > > without filter > > > > show jobs > > > > Yes > > > > without filter > > > > We can see current flink many sql statements only support showing with > full > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > Support or Not > > > > describe database > > > > No > > > > describe table > > > > Yes > > > > describe function > > > > No > > > > describe query > > > > No > > > > current flink only supports describing tables. > > > > Also, please let me know if there is a mistake. Looking forward to your > > reply. > > > > > > Best Regards, > > Ran Tao > > https://github.com/chucheng92 > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Thank you, I think this is worth a FLIP design doc to discuss the detailed syntax. Could you prepare a FLIP for that? Best, Jark On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > Hi, Jark. thanks. I have added a google doc. > > > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > > Jark Wu 于2023年2月21日周二 19:27写道： > > > Hi Ran, > > > > I think it’s always nice to support new syntax if it’s useful for the > > users. > > From my side, your syntax table is broken. Could you share it with a > > Google doc or create a JIRA issue? > > > > Best, > > Jark > > > > > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > > > Hi guys. When I recently used flink sql to manage internal metadata > > > (catalog/databases/table/functions), > > > I found that many current flink sql statements do not support filtering > > or > > > some advanced syntax, however these abilities are very useful to > > end-users. > > > > > > These are some statements I have collected so far, which are supported > on > > > other big data engines, such as spark, hive or presto. I wonder if we > can > > > support these abilities? > > > > > > In addition, the subject of this email is named 'Auxiliary Statements' > > > mainly because the alignment of these statements will not have much > > impact > > > on the core SQL runtime. > > > > > > Support or Not > > > > > > With Advanced Syntax Or Not (in/from or like) > > > > > > show create table > > > > > > Yes > > > > > > Yes > > > > > > show tables > > > > > > Yes > > > > > > Yes > > > > > > show columns > > > > > > Yes > > > > > > Yes > > > > > > show catalogs > > > > > > Yes > > > > > > without filter > > > > > > show databases > > > > > > Yes > > > > > > without filter > > > > > > show functions > > > > > > Yes > > > > > > without filter > > > > > > show views > > > > > > Yes > > > > > > without filter > > > > > > show modules > > > > > > Yes > > > > > > without filter > > > > > > show jars > > > > > > Yes > > > > > > without filter > > > > > > show jobs > > > > > > Yes > > > > > > without filter > > > > > > We can see current flink many sql statements only support showing with > > full > > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > > > Support or Not > > > > > > describe database > > > > > > No > > > > > > describe table > > > > > > Yes > > > > > > describe function > > > > > > No > > > > > > describe query > > > > > > No > > > > > > current flink only supports describing tables. > > > > > > Also, please let me know if there is a mistake. Looking forward to your > > > reply. > > > > > > > > > Best Regards, > > > Ran Tao > > > https://github.com/chucheng92 > > > > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Thanks. I will create a FLIP to illustrate some details. Best Regards, Ran Tao https://github.com/chucheng92 Jark Wu 于2023年2月21日周二 20:03写道： > Thank you, > > I think this is worth a FLIP design doc to discuss the detailed syntax. > Could you prepare a FLIP for that? > > Best, > Jark > > On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > > > Hi, Jark. thanks. I have added a google doc. > > > > > > > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > > > > Jark Wu 于2023年2月21日周二 19:27写道： > > > > > Hi Ran, > > > > > > I think it’s always nice to support new syntax if it’s useful for the > > > users. > > > From my side, your syntax table is broken. Could you share it with a > > > Google doc or create a JIRA issue? > > > > > > Best, > > > Jark > > > > > > > > > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > > > > > Hi guys. When I recently used flink sql to manage internal metadata > > > > (catalog/databases/table/functions), > > > > I found that many current flink sql statements do not support > filtering > > > or > > > > some advanced syntax, however these abilities are very useful to > > > end-users. > > > > > > > > These are some statements I have collected so far, which are > supported > > on > > > > other big data engines, such as spark, hive or presto. I wonder if we > > can > > > > support these abilities? > > > > > > > > In addition, the subject of this email is named 'Auxiliary > Statements' > > > > mainly because the alignment of these statements will not have much > > > impact > > > > on the core SQL runtime. > > > > > > > > Support or Not > > > > > > > > With Advanced Syntax Or Not (in/from or like) > > > > > > > > show create table > > > > > > > > Yes > > > > > > > > Yes > > > > > > > > show tables > > > > > > > > Yes > > > > > > > > Yes > > > > > > > > show columns > > > > > > > > Yes > > > > > > > > Yes > > > > > > > > show catalogs > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show databases > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show functions > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show views > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show modules > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show jars > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > show jobs > > > > > > > > Yes > > > > > > > > without filter > > > > > > > > We can see current flink many sql statements only support showing > with > > > full > > > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > > > > > Support or Not > > > > > > > > describe database > > > > > > > > No > > > > > > > > describe table > > > > > > > > Yes > > > > > > > > describe function > > > > > > > > No > > > > > > > > describe query > > > > > > > > No > > > > > > > > current flink only supports describing tables. > > > > > > > > Also, please let me know if there is a mistake. Looking forward to > your > > > > reply. > > > > > > > > > > > > Best Regards, > > > > Ran Tao > > > > https://github.com/chucheng92 > > > > > > > > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
+ 1 for a new FLIP It would be great if we could take this opportunity to go through all current Flink SQL syntax and define a feasible classification in the FLIP. In my opinion, the 4 categories, i.e. DDL, DML, DQL, DAL , used in Spark doc [1] make sense. We will then have a better big picture to know the gap between Flink and other big data engines like Spark and what should be done. Best regards, Jing [1] https://spark.apache.org/docs/latest/sql-ref-syntax.html#auxiliary-statements On Tue, Feb 21, 2023 at 1:15 PM Ran Tao wrote: > Thanks. I will create a FLIP to illustrate some details. > > > Best Regards, > Ran Tao > https://github.com/chucheng92 > > > Jark Wu 于2023年2月21日周二 20:03写道： > > > Thank you, > > > > I think this is worth a FLIP design doc to discuss the detailed syntax. > > Could you prepare a FLIP for that? > > > > Best, > > Jark > > > > On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > > > > > Hi, Jark. thanks. I have added a google doc. > > > > > > > > > > > > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > > > > > > Jark Wu 于2023年2月21日周二 19:27写道： > > > > > > > Hi Ran, > > > > > > > > I think it’s always nice to support new syntax if it’s useful for the > > > > users. > > > > From my side, your syntax table is broken. Could you share it with a > > > > Google doc or create a JIRA issue? > > > > > > > > Best, > > > > Jark > > > > > > > > > > > > > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > > > > > > > Hi guys. When I recently used flink sql to manage internal metadata > > > > > (catalog/databases/table/functions), > > > > > I found that many current flink sql statements do not support > > filtering > > > > or > > > > > some advanced syntax, however these abilities are very useful to > > > > end-users. > > > > > > > > > > These are some statements I have collected so far, which are > > supported > > > on > > > > > other big data engines, such as spark, hive or presto. I wonder if > we > > > can > > > > > support these abilities? > > > > > > > > > > In addition, the subject of this email is named 'Auxiliary > > Statements' > > > > > mainly because the alignment of these statements will not have much > > > > impact > > > > > on the core SQL runtime. > > > > > > > > > > Support or Not > > > > > > > > > > With Advanced Syntax Or Not (in/from or like) > > > > > > > > > > show create table > > > > > > > > > > Yes > > > > > > > > > > Yes > > > > > > > > > > show tables > > > > > > > > > > Yes > > > > > > > > > > Yes > > > > > > > > > > show columns > > > > > > > > > > Yes > > > > > > > > > > Yes > > > > > > > > > > show catalogs > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show databases > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show functions > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show views > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show modules > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show jars > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > show jobs > > > > > > > > > > Yes > > > > > > > > > > without filter > > > > > > > > > > We can see current flink many sql statements only support showing > > with > > > > full > > > > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > > > > > > > Support or Not > > > > > > > > > > describe database > > > > > > > > > > No > > > > > > > > > > describe table > > > > > > > > > > Yes > > > > > > > > > > describe function > > > > > > > > > > No > > > > > > > > > > describe query > > > > > > > > > > No > > > > > > > > > > current flink only supports describing tables. > > > > > > > > > > Also, please let me know if there is a mistake. Looking forward to > > your > > > > > reply. > > > > > > > > > > > > > > > Best Regards, > > > > > Ran Tao > > > > > https://github.com/chucheng92 > > > > > > > > > > > > > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Yes, I totally agree with your point. The current flink sql syntax is tiled together without classification. Let me try to summarize these four categories, not just Auxiliary Statements. And compare it with other popular engines. Thank you, Jing ! Best Regards, Ran Tao https://github.com/chucheng92 Jing Ge 于2023年2月21日周二 20:50写道： > + 1 for a new FLIP > > It would be great if we could take this opportunity to go through all > current Flink SQL syntax and define a feasible classification in the FLIP. > In my opinion, the 4 categories, i.e. DDL, DML, DQL, DAL , used in Spark > doc [1] make sense. > We will then have a better big picture to know the gap between Flink and > other big data engines like Spark and what should be done. > > Best regards, > Jing > > [1] > > https://spark.apache.org/docs/latest/sql-ref-syntax.html#auxiliary-statements > > On Tue, Feb 21, 2023 at 1:15 PM Ran Tao wrote: > > > Thanks. I will create a FLIP to illustrate some details. > > > > > > Best Regards, > > Ran Tao > > https://github.com/chucheng92 > > > > > > Jark Wu 于2023年2月21日周二 20:03写道： > > > > > Thank you, > > > > > > I think this is worth a FLIP design doc to discuss the detailed syntax. > > > Could you prepare a FLIP for that? > > > > > > Best, > > > Jark > > > > > > On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > > > > > > > Hi, Jark. thanks. I have added a google doc. > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > > > > > > > > Jark Wu 于2023年2月21日周二 19:27写道： > > > > > > > > > Hi Ran, > > > > > > > > > > I think it’s always nice to support new syntax if it’s useful for > the > > > > > users. > > > > > From my side, your syntax table is broken. Could you share it with > a > > > > > Google doc or create a JIRA issue? > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > > > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > > > > > > > > > Hi guys. When I recently used flink sql to manage internal > metadata > > > > > > (catalog/databases/table/functions), > > > > > > I found that many current flink sql statements do not support > > > filtering > > > > > or > > > > > > some advanced syntax, however these abilities are very useful to > > > > > end-users. > > > > > > > > > > > > These are some statements I have collected so far, which are > > > supported > > > > on > > > > > > other big data engines, such as spark, hive or presto. I wonder > if > > we > > > > can > > > > > > support these abilities? > > > > > > > > > > > > In addition, the subject of this email is named 'Auxiliary > > > Statements' > > > > > > mainly because the alignment of these statements will not have > much > > > > > impact > > > > > > on the core SQL runtime. > > > > > > > > > > > > Support or Not > > > > > > > > > > > > With Advanced Syntax Or Not (in/from or like) > > > > > > > > > > > > show create table > > > > > > > > > > > > Yes > > > > > > > > > > > > Yes > > > > > > > > > > > > show tables > > > > > > > > > > > > Yes > > > > > > > > > > > > Yes > > > > > > > > > > > > show columns > > > > > > > > > > > > Yes > > > > > > > > > > > > Yes > > > > > > > > > > > > show catalogs > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show databases > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show functions > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show views > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show modules > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show jars > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > show jobs > > > > > > > > > > > > Yes > > > > > > > > > > > > without filter > > > > > > > > > > > > We can see current flink many sql statements only support showing > > > with > > > > > full > > > > > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > > > > > > > > > Support or Not > > > > > > > > > > > > describe database > > > > > > > > > > > > No > > > > > > > > > > > > describe table > > > > > > > > > > > > Yes > > > > > > > > > > > > describe function > > > > > > > > > > > > No > > > > > > > > > > > > describe query > > > > > > > > > > > > No > > > > > > > > > > > > current flink only supports describing tables. > > > > > > > > > > > > Also, please let me know if there is a mistake. Looking forward > to > > > your > > > > > > reply. > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > Ran Tao > > > > > > https://github.com/chucheng92 > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi Ran, When you're creating your FLIP, please also have a look at Calcite SQL parser, since that's used by Flink [1] Best regards, Martijn [1] https://calcite.apache.org/docs/reference.html On Tue, Feb 21, 2023 at 2:10 PM Ran Tao wrote: > Yes, I totally agree with your point. The current flink sql syntax is tiled > together without classification. > Let me try to summarize these four categories, not just Auxiliary > Statements. And compare it with other popular engines. > > Thank you, Jing ! > > Best Regards, > Ran Tao > https://github.com/chucheng92 > > > Jing Ge 于2023年2月21日周二 20:50写道： > > > + 1 for a new FLIP > > > > It would be great if we could take this opportunity to go through all > > current Flink SQL syntax and define a feasible classification in the > FLIP. > > In my opinion, the 4 categories, i.e. DDL, DML, DQL, DAL , used in Spark > > doc [1] make sense. > > We will then have a better big picture to know the gap between Flink and > > other big data engines like Spark and what should be done. > > > > Best regards, > > Jing > > > > [1] > > > > > https://spark.apache.org/docs/latest/sql-ref-syntax.html#auxiliary-statements > > > > On Tue, Feb 21, 2023 at 1:15 PM Ran Tao wrote: > > > > > Thanks. I will create a FLIP to illustrate some details. > > > > > > > > > Best Regards, > > > Ran Tao > > > https://github.com/chucheng92 > > > > > > > > > Jark Wu 于2023年2月21日周二 20:03写道： > > > > > > > Thank you, > > > > > > > > I think this is worth a FLIP design doc to discuss the detailed > syntax. > > > > Could you prepare a FLIP for that? > > > > > > > > Best, > > > > Jark > > > > > > > > On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > > > > > > > > > Hi, Jark. thanks. I have added a google doc. > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > > > > > > > > > > Jark Wu 于2023年2月21日周二 19:27写道： > > > > > > > > > > > Hi Ran, > > > > > > > > > > > > I think it’s always nice to support new syntax if it’s useful for > > the > > > > > > users. > > > > > > From my side, your syntax table is broken. Could you share it > with > > a > > > > > > Google doc or create a JIRA issue? > > > > > > > > > > > > Best, > > > > > > Jark > > > > > > > > > > > > > > > > > > > > > > > > > 2023年2月21日 17:51，Ran Tao 写道： > > > > > > > > > > > > > > Hi guys. When I recently used flink sql to manage internal > > metadata > > > > > > > (catalog/databases/table/functions), > > > > > > > I found that many current flink sql statements do not support > > > > filtering > > > > > > or > > > > > > > some advanced syntax, however these abilities are very useful > to > > > > > > end-users. > > > > > > > > > > > > > > These are some statements I have collected so far, which are > > > > supported > > > > > on > > > > > > > other big data engines, such as spark, hive or presto. I wonder > > if > > > we > > > > > can > > > > > > > support these abilities? > > > > > > > > > > > > > > In addition, the subject of this email is named 'Auxiliary > > > > Statements' > > > > > > > mainly because the alignment of these statements will not have > > much > > > > > > impact > > > > > > > on the core SQL runtime. > > > > > > > > > > > > > > Support or Not > > > > > > > > > > > > > > With Advanced Syntax Or Not (in/from or like) > > > > > > > > > > > > > > show create table > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > show tables > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > show columns > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > show catalogs > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show databases > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show functions > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show views > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show modules > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show jars > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > show jobs > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > without filter > > > > > > > > > > > > > > We can see current flink many sql statements only support > showing > > > > with > > > > > > full > > > > > > > datas, without 'FROM/IN' or 'LIKE' filter clause. > > > > > > > > > > > > > > Support or Not > > > > > > > > > > > > > > describe database > > > > > > > > > > > > > > No > > > > > > > > > > > > > > describe table > > > > > > > > > > > > > > Yes > > > > > > > > > > > > > > describe function > > > > > > > > > > > > > > No > > > > > > > > > > > > > > describe query > > > > > > > > > > > > > >
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi Ran, adding additional filter and selection capabilities makes sense. However, I would recommend to not only check Spark or Hive but also focus on bigger commercial players such as Snowflake, Oracle, SQL Server. In any case, maybe we should also investigate more thoughts on a Flink INFORMATION_SCHEMA as the SQL standard defines it. Having a queryable table that contains metadata on which we can run arbitrary SQL logic not only filtering but aggregations like counting or projections. This would avoid touching the parser everytime and introduce more and more SQL keywords. What do you think? Regards, Timo On 21.02.23 14:32, Martijn Visser wrote: Hi Ran, When you're creating your FLIP, please also have a look at Calcite SQL parser, since that's used by Flink [1] Best regards, Martijn [1] https://calcite.apache.org/docs/reference.html On Tue, Feb 21, 2023 at 2:10 PM Ran Tao wrote: Yes, I totally agree with your point. The current flink sql syntax is tiled together without classification. Let me try to summarize these four categories, not just Auxiliary Statements. And compare it with other popular engines. Thank you, Jing ! Best Regards, Ran Tao https://github.com/chucheng92 Jing Ge 于2023年2月21日周二 20:50写道： + 1 for a new FLIP It would be great if we could take this opportunity to go through all current Flink SQL syntax and define a feasible classification in the FLIP. In my opinion, the 4 categories, i.e. DDL, DML, DQL, DAL , used in Spark doc [1] make sense. We will then have a better big picture to know the gap between Flink and other big data engines like Spark and what should be done. Best regards, Jing [1] https://spark.apache.org/docs/latest/sql-ref-syntax.html#auxiliary-statements On Tue, Feb 21, 2023 at 1:15 PM Ran Tao wrote: Thanks. I will create a FLIP to illustrate some details. Best Regards, Ran Tao https://github.com/chucheng92 Jark Wu 于2023年2月21日周二 20:03写道： Thank you, I think this is worth a FLIP design doc to discuss the detailed syntax. Could you prepare a FLIP for that? Best, Jark On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: Hi, Jark. thanks. I have added a google doc. https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing Jark Wu 于2023年2月21日周二 19:27写道： Hi Ran, I think it’s always nice to support new syntax if it’s useful for the users. From my side, your syntax table is broken. Could you share it with a Google doc or create a JIRA issue? Best, Jark 2023年2月21日 17:51，Ran Tao 写道： Hi guys. When I recently used flink sql to manage internal metadata (catalog/databases/table/functions), I found that many current flink sql statements do not support filtering or some advanced syntax, however these abilities are very useful to end-users. These are some statements I have collected so far, which are supported on other big data engines, such as spark, hive or presto. I wonder if we can support these abilities? In addition, the subject of this email is named 'Auxiliary Statements' mainly because the alignment of these statements will not have much impact on the core SQL runtime. Support or Not With Advanced Syntax Or Not (in/from or like) show create table Yes Yes show tables Yes Yes show columns Yes Yes show catalogs Yes without filter show databases Yes without filter show functions Yes without filter show views Yes without filter show modules Yes without filter show jars Yes without filter show jobs Yes without filter We can see current flink many sql statements only support showing with full datas, without 'FROM/IN' or 'LIKE' filter clause. Support or Not describe database No describe table Yes describe function No describe query No current flink only supports describing tables. Also, please let me know if there is a mistake. Looking forward to your reply. Best Regards, Ran Tao https://github.com/chucheng92
Re: [DISCUSS] Proposal to improve Auxiliary Sql Statements
Hi Timo, Thanks. yes. We need to make Flink's implementation of this feature more scalable and versatile. I will try to update this FLIP on the comparison and difference of the implementation of some features on other popular or commercial engines you mentioned. Let us see how to find a better way to solve this problem to avoid subsequent frequent changes to the parser module and keywords. Best Regards, Ran Tao https://github.com/chucheng92 Timo Walther 于2023年2月21日周二 22:58写道： > Hi Ran, > > adding additional filter and selection capabilities makes sense. > However, I would recommend to not only check Spark or Hive but also > focus on bigger commercial players such as Snowflake, Oracle, SQL Server. > > In any case, maybe we should also investigate more thoughts on a Flink > INFORMATION_SCHEMA as the SQL standard defines it. Having a queryable > table that contains metadata on which we can run arbitrary SQL logic not > only filtering but aggregations like counting or projections. > > This would avoid touching the parser everytime and introduce more and > more SQL keywords. > > What do you think? > > Regards, > Timo > > On 21.02.23 14:32, Martijn Visser wrote: > > Hi Ran, > > > > When you're creating your FLIP, please also have a look at Calcite SQL > > parser, since that's used by Flink [1] > > > > Best regards, > > > > Martijn > > > > [1] https://calcite.apache.org/docs/reference.html > > > > On Tue, Feb 21, 2023 at 2:10 PM Ran Tao wrote: > > > >> Yes, I totally agree with your point. The current flink sql syntax is > tiled > >> together without classification. > >> Let me try to summarize these four categories, not just Auxiliary > >> Statements. And compare it with other popular engines. > >> > >> Thank you, Jing ! > >> > >> Best Regards, > >> Ran Tao > >> https://github.com/chucheng92 > >> > >> > >> Jing Ge 于2023年2月21日周二 20:50写道： > >> > >>> + 1 for a new FLIP > >>> > >>> It would be great if we could take this opportunity to go through all > >>> current Flink SQL syntax and define a feasible classification in the > >> FLIP. > >>> In my opinion, the 4 categories, i.e. DDL, DML, DQL, DAL , used in > Spark > >>> doc [1] make sense. > >>> We will then have a better big picture to know the gap between Flink > and > >>> other big data engines like Spark and what should be done. > >>> > >>> Best regards, > >>> Jing > >>> > >>> [1] > >>> > >>> > >> > https://spark.apache.org/docs/latest/sql-ref-syntax.html#auxiliary-statements > >>> > >>> On Tue, Feb 21, 2023 at 1:15 PM Ran Tao wrote: > >>> > Thanks. I will create a FLIP to illustrate some details. > > > Best Regards, > Ran Tao > https://github.com/chucheng92 > > > Jark Wu 于2023年2月21日周二 20:03写道： > > > Thank you, > > > > I think this is worth a FLIP design doc to discuss the detailed > >> syntax. > > Could you prepare a FLIP for that? > > > > Best, > > Jark > > > > On Tue, 21 Feb 2023 at 19:37, Ran Tao wrote: > > > >> Hi, Jark. thanks. I have added a google doc. > >> > >> > >> > > > > >>> > >> > https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/edit?usp=sharing > >> > >> Jark Wu 于2023年2月21日周二 19:27写道： > >> > >>> Hi Ran, > >>> > >>> I think it’s always nice to support new syntax if it’s useful for > >>> the > >>> users. > >>> From my side, your syntax table is broken. Could you share it > >> with > >>> a > >>> Google doc or create a JIRA issue? > >>> > >>> Best, > >>> Jark > >>> > >>> > >>> > 2023年2月21日 17:51，Ran Tao 写道： > > Hi guys. When I recently used flink sql to manage internal > >>> metadata > (catalog/databases/table/functions), > I found that many current flink sql statements do not support > > filtering > >>> or > some advanced syntax, however these abilities are very useful > >> to > >>> end-users. > > These are some statements I have collected so far, which are > > supported > >> on > other big data engines, such as spark, hive or presto. I wonder > >>> if > we > >> can > support these abilities? > > In addition, the subject of this email is named 'Auxiliary > > Statements' > mainly because the alignment of these statements will not have > >>> much > >>> impact > on the core SQL runtime. > > Support or Not > > With Advanced Syntax Or Not (in/from or like) > > show create table > > Yes > > Yes > > show tables > > Yes > > Yes > > show columns > > Yes > > Yes > > show catalogs > > Yes > > without filter > > show d
