[jira] [Created] (FLINK-35406) Use inner serializer when casting RAW type to BINARY or STRING in cast rules
Zhenghua Gao created FLINK-35406: Summary: Use inner serializer when casting RAW type to BINARY or STRING in cast rules Key: FLINK-35406 URL: https://issues.apache.org/jira/browse/FLINK-35406 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.19.0 Reporter: Zhenghua Gao The generated code in RawToStringCastRule and RawToBinaryCastRule use BinaryRawValueData::toBytes and BinaryRawValueData::toObject to convert RawValueData(to java object or byte array), which should use inner serializer instead of RawValueDataSerializer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-21026) Align column list specification with Hive in INSERT statement
Zhenghua Gao created FLINK-21026:
Summary: Align column list specification with Hive in INSERT
statement
Key: FLINK-21026
URL: https://issues.apache.org/jira/browse/FLINK-21026
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Zhenghua Gao
[HIVE-9481|https://issues.apache.org/jira/browse/HIVE-9481] allows column list
specification in INSERT statement. The syntax is:
{code:java}
INSERT INTO TABLE table_name
[PARTITION (partcol1[=val1], partcol2[=val2] ...)]
[(column list)]
select_statement FROM from_statement
{code}
In the MeanWhile, flink introduces PARTITION syntax that the PARTITION clause
appears after the COLUMN LIST clause. It looks weird and luckily we don't
support COLUMN LIST clause
now[FLINK-18726|https://issues.apache.org/jira/browse/FLINK-18726]. I think
it'a good change to align this with Hive now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17113) Refactor view support in SQL Client
Zhenghua Gao created FLINK-17113: Summary: Refactor view support in SQL Client Key: FLINK-17113 URL: https://issues.apache.org/jira/browse/FLINK-17113 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17112) Support DESCRIBE VIEW view_name in Flink SQL
Zhenghua Gao created FLINK-17112: Summary: Support DESCRIBE VIEW view_name in Flink SQL Key: FLINK-17112 URL: https://issues.apache.org/jira/browse/FLINK-17112 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17111) Support SHOW VIEWS in Flink SQL
Zhenghua Gao created FLINK-17111: Summary: Support SHOW VIEWS in Flink SQL Key: FLINK-17111 URL: https://issues.apache.org/jira/browse/FLINK-17111 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17106) Support create/drop view in Flink SQL
Zhenghua Gao created FLINK-17106: Summary: Support create/drop view in Flink SQL Key: FLINK-17106 URL: https://issues.apache.org/jira/browse/FLINK-17106 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17105) FLIP-71: E2E viewsupport
Zhenghua Gao created FLINK-17105: Summary: FLIP-71: E2E viewsupport Key: FLINK-17105 URL: https://issues.apache.org/jira/browse/FLINK-17105 Project: Flink Issue Type: New Feature Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17098) CatalogManager#dropTemporaryTable and dropTemporaryView should use ObjectIdentifier as its argument
Zhenghua Gao created FLINK-17098: Summary: CatalogManager#dropTemporaryTable and dropTemporaryView should use ObjectIdentifier as its argument Key: FLINK-17098 URL: https://issues.apache.org/jira/browse/FLINK-17098 Project: Flink Issue Type: Improvement Components: Table SQL / API Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Since CatalogManager#createTable, createTemporaryTable and dropTable use the given fully qualified ObjectIdentifier to create or drop tables/temporary tables, we should also use ObjectIdentifier (instead of UnresolvedIdentifier) in dropTemporaryTable and dropTemporaryView. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] FLIP-71: E2E View support in Flink SQL
Hi everyone, Thanks for the votes. So far, we have - 3 binding +1 votes (Timo, Jingsong, Jark) - 4 non-binding +1 votes (Danny, zoudan, Benchao, godfrey) - no -1 vote The voting time has past and there is enough +1 votes to consider the FLIP-71 approved. Thanks you all. *Best Regards,* *Zhenghua Gao* On Mon, Apr 13, 2020 at 10:30 AM Jark Wu wrote: > +1 > > Best, > Jark > > On Sun, 12 Apr 2020 at 12:28, Benchao Li wrote: > > > +1 (non-binding) > > > > zoudan 于2020年4月12日周日 上午9:52写道： > > > > > +1 (non-binding) > > > > > > Best, > > > Dan Zou > > > > > > > > > > 在 2020年4月10日，09:30，Danny Chan 写道： > > > > > > > > +1 from my side. > > > > > > > > Best, > > > > Danny Chan > > > > 在 2020年4月9日 +0800 PM9:23，Timo Walther ，写道： > > > >> +1 (binding) > > > >> > > > >> Thanks for your efforts. > > > >> > > > >> Regards, > > > >> Timo > > > >> > > > >> > > > >> On 09.04.20 14:46, Zhenghua Gao wrote: > > > >>> Hi all, > > > >>> > > > >>> I'd like to start the vote for FLIP-71[1] which adds E2E view > support > > > in > > > >>> Flink SQL. > > > >>> This FLIP is discussed in the thread[2]. > > > >>> > > > >>> The vote will be open for at least 72 hours. Unless there is an > > > objection. > > > >>> I will try to > > > >>> close it by April 13, 2020 09:00 UTC if we have received sufficient > > > votes. > > > >>> > > > >>> [1] > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL > > > >>> > > > >>> [2] > > > >>> > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787 > > > >>> > > > >>> *Best Regards,* > > > >>> *Zhenghua Gao* > > > >>> > > > >> > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: libenc...@gmail.com; libenc...@pku.edu.cn > > >
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
Hi Jark, >> Shall we remove the view support in those commands if we want to support a >> dedicate "SHOW VIEWS|DESCRIBE VIEW name"? Yes, we should correct those commands in SQL client. Will open tickets after the vote. *Best Regards,* *Zhenghua Gao* On Sat, Apr 11, 2020 at 11:25 AM Jark Wu wrote: > Sorry for the late reply, > > I have some concern around "Supporting SHOW VIEWS|DESCRIBE VIEW name". > Currently, in SQL CLI, the "SHOW TABLES" will also list views and "DESCRIBE > name" can also describe a view. > Shall we remove the view support in those commands if we want to support a > dedicate "SHOW VIEWS|DESCRIBE VIEW name"? > > Brest, > Jark > > On Wed, 8 Apr 2020 at 23:49, Timo Walther wrote: > > > I didn't know that. We should definitely implement this asap. Please > > open a JIRA issue. > > > > Thanks, > > Timo > > > > > > On 08.04.20 14:29, Zhenghua Gao wrote: > > > Hi Timo, > > > > > > Actually "TEMPORARY" is not supported in table DDL now. > > > But you are right I could support "CREATE TEMPORARY VIEW" in this FLIP. > > > And may be we should open a separate JIRA ticket to track supporting it > > in > > > table DDL？ > > > > > > *Best Regards,* > > > *Zhenghua Gao* > > > > > > > > > On Wed, Apr 8, 2020 at 7:48 PM Timo Walther > wrote: > > > > > >> Hi Zhenghua, > > >> > > >> FLINK-10232 is quite old and a lot of stuff was discussed and agreed > on > > >> since then. I don't like to postpone the 'TEMPORARY' keyword because > it > > >> is a important concept that is already part of the Table API (see > > >> TableEnvironment.createTemporaryView) and in function DDL and table > DDL. > > >> It is not complicated to supported it in this FLIP and just a couple > of > > >> line of code more. > > >> > > >> Regards, > > >> Timo > > >> > > >> On 08.04.20 11:27, Zhenghua Gao wrote: > > >>> Another concern about "CREATE DDL" is: > > >>> > > >>> FLINK-10232 proposes using "IF NOT EXISTS" to control the behavior > > when a > > >>> view or table with the same name already exists. > > >>> And "OR REPLACE" for type/library/function DDL. > > >>> > > >>> @godfrey he I will keep the "IF NOT EXISTS" > > syntax > > >>> and postpone the "OR REPLACE" syntax until we need it. > > >>> > > >>> > > >>> *Best Regards,* > > >>> *Zhenghua Gao* > > >>> > > >>> > > >>> On Wed, Apr 8, 2020 at 4:46 PM Zhenghua Gao > wrote: > > >>> > > >>>> Hi Timo, > > >>>> > > >>>> Shall we postpone the support of 'TEMPORARY' keyword since it's not > > >>>> mentioned in FLINK-10232? > > >>>> <https://issues.apache.org/jira/browse/FLINK-10232> > > >>>> > > >>>> *Best Regards,* > > >>>> *Zhenghua Gao* > > >>>> > > >>>> > > >>>> On Wed, Apr 8, 2020 at 3:30 PM Timo Walther > > wrote: > > >>>> > > >>>>> Hi Zhenghua, > > >>>>> > > >>>>> VIEWS should also support the TEMPORARY keyword according to > FLIP-64. > > >>>>> > > >>>>> Otherwise the FLIP looks good to me. > > >>>>> > > >>>>> Regards, > > >>>>> Timo > > >>>>> > > >>>>> > > >>>>> On 08.04.20 07:31, Zhenghua Gao wrote: > > >>>>>> @Danny Chan you‘re right. I have updated > > the > > >>>>> doc. > > >>>>>> > > >>>>>> *Best Regards,* > > >>>>>> *Zhenghua Gao* > > >>>>>> > > >>>>>> > > >>>>>> On Wed, Apr 8, 2020 at 1:20 PM Danny Chan > > >> wrote: > > >>>>>> > > >>>>>>> +1 for the proposal, a small concern for drop view statement: > > >>>>>>> > > >>>>>>> dropViewStatement: > > >>>>>>> DROP VIEW name [ IF EXI
[VOTE] FLIP-71: E2E View support in Flink SQL
Hi all, I'd like to start the vote for FLIP-71[1] which adds E2E view support in Flink SQL. This FLIP is discussed in the thread[2]. The vote will be open for at least 72 hours. Unless there is an objection. I will try to close it by April 13, 2020 09:00 UTC if we have received sufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787 *Best Regards,* *Zhenghua Gao*
[jira] [Created] (FLINK-17067) CatalogManager#createTable and createTemporaryTable should provide consistent semantics
Zhenghua Gao created FLINK-17067: Summary: CatalogManager#createTable and createTemporaryTable should provide consistent semantics Key: FLINK-17067 URL: https://issues.apache.org/jira/browse/FLINK-17067 Project: Flink Issue Type: Improvement Components: Table SQL / API Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Currently CatalogManager#createTable provides [IF NOT EXISTS] semantic and CatalogManager#createTemporaryTable provides [OR REPLACE] semantic. IMO they should provide consistent semantics: [IF NOT EXISTS] or [OR REPLACE] or BOTH. I prefer [IF NOT EXISTS] since we didn't support [OR REPLACE] in table DDL(and view DDL) currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
Hi Timo, Actually "TEMPORARY" is not supported in table DDL now. But you are right I could support "CREATE TEMPORARY VIEW" in this FLIP. And may be we should open a separate JIRA ticket to track supporting it in table DDL？ *Best Regards,* *Zhenghua Gao* On Wed, Apr 8, 2020 at 7:48 PM Timo Walther wrote: > Hi Zhenghua, > > FLINK-10232 is quite old and a lot of stuff was discussed and agreed on > since then. I don't like to postpone the 'TEMPORARY' keyword because it > is a important concept that is already part of the Table API (see > TableEnvironment.createTemporaryView) and in function DDL and table DDL. > It is not complicated to supported it in this FLIP and just a couple of > line of code more. > > Regards, > Timo > > On 08.04.20 11:27, Zhenghua Gao wrote: > > Another concern about "CREATE DDL" is: > > > > FLINK-10232 proposes using "IF NOT EXISTS" to control the behavior when a > > view or table with the same name already exists. > > And "OR REPLACE" for type/library/function DDL. > > > > @godfrey he I will keep the "IF NOT EXISTS" syntax > > and postpone the "OR REPLACE" syntax until we need it. > > > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Wed, Apr 8, 2020 at 4:46 PM Zhenghua Gao wrote: > > > >> Hi Timo, > >> > >> Shall we postpone the support of 'TEMPORARY' keyword since it's not > >> mentioned in FLINK-10232? > >> <https://issues.apache.org/jira/browse/FLINK-10232> > >> > >> *Best Regards,* > >> *Zhenghua Gao* > >> > >> > >> On Wed, Apr 8, 2020 at 3:30 PM Timo Walther wrote: > >> > >>> Hi Zhenghua, > >>> > >>> VIEWS should also support the TEMPORARY keyword according to FLIP-64. > >>> > >>> Otherwise the FLIP looks good to me. > >>> > >>> Regards, > >>> Timo > >>> > >>> > >>> On 08.04.20 07:31, Zhenghua Gao wrote: > >>>> @Danny Chan you‘re right. I have updated the > >>> doc. > >>>> > >>>> *Best Regards,* > >>>> *Zhenghua Gao* > >>>> > >>>> > >>>> On Wed, Apr 8, 2020 at 1:20 PM Danny Chan > wrote: > >>>> > >>>>> +1 for the proposal, a small concern for drop view statement: > >>>>> > >>>>> dropViewStatement: > >>>>> DROP VIEW name [ IF EXISTS ] > >>>>> I think the drop statement should be > >>>>> DROP VIEW [ IF EXISTS ] name > >>>>> > >>>>> Best, > >>>>> Danny Chan > >>>>> 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道： > >>>>>> This FLIP seems to be quite straightforward, +1 from my side. > >>>>>> > >>>>>> Best, > >>>>>> Kurt > >>>>>> > >>>>>> > >>>>>> On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao > wrote: > >>>>>> > >>>>>>> forward the reply to ML too. > >>>>>>> > >>>>>>> > >>>>>>> *Best Regards,* > >>>>>>> *Zhenghua Gao* > >>>>>>> > >>>>>>> > >>>>>>> -- Forwarded message - > >>>>>>> From: Zhenghua Gao > >>>>>>> Date: Tue, Apr 7, 2020 at 8:40 PM > >>>>>>> Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL > >>>>>>> To: godfrey he > >>>>>>> > >>>>>>> > >>>>>>>>> regarding to "Interoperability between Flink and Hive is not > >>>>>>> guaranteed", can you explain this more? > >>>>>>> We have several limitations of interoperability between flink > objects > >>>>> and > >>>>>>> hive objects (tables, functions, etc). > >>>>>>> So we don't promise the interoperability of views between flink and > >>>>> hive > >>>>>>> since a view is defined base on these objects. > >>>>>>> > >>>>>>>>> "CREATE VIEW [ IF NOT EXISTS ]" > >>>>>>> This should be "CREATE VIEW [OR REPLACE]&quo
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
Another concern about "CREATE DDL" is: FLINK-10232 proposes using "IF NOT EXISTS" to control the behavior when a view or table with the same name already exists. And "OR REPLACE" for type/library/function DDL. @godfrey he I will keep the "IF NOT EXISTS" syntax and postpone the "OR REPLACE" syntax until we need it. *Best Regards,* *Zhenghua Gao* On Wed, Apr 8, 2020 at 4:46 PM Zhenghua Gao wrote: > Hi Timo, > > Shall we postpone the support of 'TEMPORARY' keyword since it's not > mentioned in FLINK-10232? > <https://issues.apache.org/jira/browse/FLINK-10232> > > *Best Regards,* > *Zhenghua Gao* > > > On Wed, Apr 8, 2020 at 3:30 PM Timo Walther wrote: > >> Hi Zhenghua, >> >> VIEWS should also support the TEMPORARY keyword according to FLIP-64. >> >> Otherwise the FLIP looks good to me. >> >> Regards, >> Timo >> >> >> On 08.04.20 07:31, Zhenghua Gao wrote: >> > @Danny Chan you‘re right. I have updated the >> doc. >> > >> > *Best Regards,* >> > *Zhenghua Gao* >> > >> > >> > On Wed, Apr 8, 2020 at 1:20 PM Danny Chan wrote: >> > >> >> +1 for the proposal, a small concern for drop view statement: >> >> >> >> dropViewStatement: >> >>DROP VIEW name [ IF EXISTS ] >> >> I think the drop statement should be >> >> DROP VIEW [ IF EXISTS ] name >> >> >> >> Best, >> >> Danny Chan >> >> 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道： >> >>> This FLIP seems to be quite straightforward, +1 from my side. >> >>> >> >>> Best, >> >>> Kurt >> >>> >> >>> >> >>> On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao wrote: >> >>> >> >>>> forward the reply to ML too. >> >>>> >> >>>> >> >>>> *Best Regards,* >> >>>> *Zhenghua Gao* >> >>>> >> >>>> >> >>>> -- Forwarded message - >> >>>> From: Zhenghua Gao >> >>>> Date: Tue, Apr 7, 2020 at 8:40 PM >> >>>> Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL >> >>>> To: godfrey he >> >>>> >> >>>> >> >>>>>> regarding to "Interoperability between Flink and Hive is not >> >>>> guaranteed", can you explain this more? >> >>>> We have several limitations of interoperability between flink objects >> >> and >> >>>> hive objects (tables, functions, etc). >> >>>> So we don't promise the interoperability of views between flink and >> >> hive >> >>>> since a view is defined base on these objects. >> >>>> >> >>>>>> "CREATE VIEW [ IF NOT EXISTS ]" >> >>>> This should be "CREATE VIEW [OR REPLACE]". >> >>>> >> >>>>>> "DESC" >> >>>> It's a shortcut of "DESCRIBE" in SQL Client (See desc table xxx). >> >>>> In DDL, we should only support "SHOW VIEWS" and "DESCRIBE VIEW xxx". >> >>>> >> >>>> I have updated the design doc, thanks. >> >>>> >> >>>> *Best Regards,* >> >>>> *Zhenghua Gao* >> >>>> >> >>>> >> >>>> On Tue, Apr 7, 2020 at 8:09 PM godfrey he >> wrote: >> >>>> >> >>>>> Hi Zhenghua, >> >>>>> >> >>>>> Thanks for driving this. It's one step forward that TableEnvironment >> >>>>> supports more complete SQLs. >> >>>>> I have a few minor questions: >> >>>>> 1. regarding to "Interoperability between Flink and Hive is not >> >>>>> guaranteed", can you explain this more? >> >>>>> 2. regarding to "The Grammar", Calcite does not support "CREATE VIEW >> >> [ IF >> >>>>> NOT EXISTS ]", instead supports "CREATE [OR REPLACE]". "SHOW VIEWS" >> >> and >> >>>>> "DESCRIBE VIEW xx" are not supported now. Calcite does not support >> >>>> describe >> >>>>> an object through "DESC" . I think It&#
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
Hi Timo, Shall we postpone the support of 'TEMPORARY' keyword since it's not mentioned in FLINK-10232? <https://issues.apache.org/jira/browse/FLINK-10232> *Best Regards,* *Zhenghua Gao* On Wed, Apr 8, 2020 at 3:30 PM Timo Walther wrote: > Hi Zhenghua, > > VIEWS should also support the TEMPORARY keyword according to FLIP-64. > > Otherwise the FLIP looks good to me. > > Regards, > Timo > > > On 08.04.20 07:31, Zhenghua Gao wrote: > > @Danny Chan you‘re right. I have updated the > doc. > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Wed, Apr 8, 2020 at 1:20 PM Danny Chan wrote: > > > >> +1 for the proposal, a small concern for drop view statement: > >> > >> dropViewStatement: > >>DROP VIEW name [ IF EXISTS ] > >> I think the drop statement should be > >> DROP VIEW [ IF EXISTS ] name > >> > >> Best, > >> Danny Chan > >> 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道： > >>> This FLIP seems to be quite straightforward, +1 from my side. > >>> > >>> Best, > >>> Kurt > >>> > >>> > >>> On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao wrote: > >>> > >>>> forward the reply to ML too. > >>>> > >>>> > >>>> *Best Regards,* > >>>> *Zhenghua Gao* > >>>> > >>>> > >>>> -- Forwarded message - > >>>> From: Zhenghua Gao > >>>> Date: Tue, Apr 7, 2020 at 8:40 PM > >>>> Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL > >>>> To: godfrey he > >>>> > >>>> > >>>>>> regarding to "Interoperability between Flink and Hive is not > >>>> guaranteed", can you explain this more? > >>>> We have several limitations of interoperability between flink objects > >> and > >>>> hive objects (tables, functions, etc). > >>>> So we don't promise the interoperability of views between flink and > >> hive > >>>> since a view is defined base on these objects. > >>>> > >>>>>> "CREATE VIEW [ IF NOT EXISTS ]" > >>>> This should be "CREATE VIEW [OR REPLACE]". > >>>> > >>>>>> "DESC" > >>>> It's a shortcut of "DESCRIBE" in SQL Client (See desc table xxx). > >>>> In DDL, we should only support "SHOW VIEWS" and "DESCRIBE VIEW xxx". > >>>> > >>>> I have updated the design doc, thanks. > >>>> > >>>> *Best Regards,* > >>>> *Zhenghua Gao* > >>>> > >>>> > >>>> On Tue, Apr 7, 2020 at 8:09 PM godfrey he > wrote: > >>>> > >>>>> Hi Zhenghua, > >>>>> > >>>>> Thanks for driving this. It's one step forward that TableEnvironment > >>>>> supports more complete SQLs. > >>>>> I have a few minor questions: > >>>>> 1. regarding to "Interoperability between Flink and Hive is not > >>>>> guaranteed", can you explain this more? > >>>>> 2. regarding to "The Grammar", Calcite does not support "CREATE VIEW > >> [ IF > >>>>> NOT EXISTS ]", instead supports "CREATE [OR REPLACE]". "SHOW VIEWS" > >> and > >>>>> "DESCRIBE VIEW xx" are not supported now. Calcite does not support > >>>> describe > >>>>> an object through "DESC" . I think It's better this flip can support > >>>> "SHOW > >>>>> VIEWS" and "DESCRIBE VIEW xx". > >>>>> > >>>>> Best, > >>>>> Godfrey > >>>>> > >>>>> Zhenghua Gao 于2020年4月3日周五 下午3:04写道： > >>>>> > >>>>>> Hi community, > >>>>>> > >>>>>> It's a long time since we started the discussion of supporting > >> view in > >>>>>> FLINK SQL. > >>>>>> Flink also continues to move forward since then. > >>>>>> FLINK-10232 introduces the grammar and FLINK-12905 supports > >> CatalogView > >>>>>> in blink planner. > >>>>>> The missing link is validate the view definition and store the > >>>>>> original/expanded text in the catalog. > >>>>>> I have updated the design doc of FLIP-71, > >>>>>> > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL > >>>>>> > >>>>>> Any comments and feedbacks are welcome and appreciated. Thanks. > >>>>>> > >>>>>> *Best Regards,* > >>>>>> *Zhenghua Gao* > >>>>>> > >>>>>> > >>>>>> On Tue, Sep 17, 2019 at 11:51 AM Zhenghua Gao > >> wrote: > >>>>>> > >>>>>>> Hi folks, > >>>>>>> > >>>>>>> In umbrella task FLINK-10232 we have introduced CREATE/DROP VIEW > >>>> grammar > >>>>>>> in our module flink-sql-parser. But we don't support view > >> objects in > >>>>>>> neither blink planner nor old planner. > >>>>>>> > >>>>>>> I'd like to kick off a discussion on end to end view support in > >> Flink > >>>>>>> SQL in blink planner. It's helpful to improve the usability of > >> the > >>>>>>> framework for SQL users. > >>>>>>> > >>>>>>> > >>>>>>> > >>>> > >> > https://docs.google.com/document/d/14bx0t8wYH7_o4ChNkDoBFGn-i0T-Q7kUiOFvDd13_Fk/edit#heading=h.m031smarjj9p > >>>>>>> > >>>>>>> In short, it: > >>>>>>> - support define views and store them in catalog > >>>>>>> - support drop view definitions from catalog > >>>>>>> - support query views > >>>>>>> - support other view related DDLs > >>>>>>> > >>>>>>> Any comments and feedbacks are welcome and appreciated. Thanks. > >>>>>>> > >>>>>>> *Best Regards,* > >>>>>>> *Zhenghua Gao* > >>>>>>> > >>>>>> > >>>> > >> > > > >
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
@Danny Chan you‘re right. I have updated the doc. *Best Regards,* *Zhenghua Gao* On Wed, Apr 8, 2020 at 1:20 PM Danny Chan wrote: > +1 for the proposal, a small concern for drop view statement: > > dropViewStatement: > DROP VIEW name [ IF EXISTS ] > I think the drop statement should be > DROP VIEW [ IF EXISTS ] name > > Best, > Danny Chan > 在 2020年4月8日 +0800 AM11:54，Kurt Young ，写道： > > This FLIP seems to be quite straightforward, +1 from my side. > > > > Best, > > Kurt > > > > > > On Tue, Apr 7, 2020 at 8:42 PM Zhenghua Gao wrote: > > > > > forward the reply to ML too. > > > > > > > > > *Best Regards,* > > > *Zhenghua Gao* > > > > > > > > > -- Forwarded message - > > > From: Zhenghua Gao > > > Date: Tue, Apr 7, 2020 at 8:40 PM > > > Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL > > > To: godfrey he > > > > > > > > > > > regarding to "Interoperability between Flink and Hive is not > > > guaranteed", can you explain this more? > > > We have several limitations of interoperability between flink objects > and > > > hive objects (tables, functions, etc). > > > So we don't promise the interoperability of views between flink and > hive > > > since a view is defined base on these objects. > > > > > > > > "CREATE VIEW [ IF NOT EXISTS ]" > > > This should be "CREATE VIEW [OR REPLACE]". > > > > > > > > "DESC" > > > It's a shortcut of "DESCRIBE" in SQL Client (See desc table xxx). > > > In DDL, we should only support "SHOW VIEWS" and "DESCRIBE VIEW xxx". > > > > > > I have updated the design doc, thanks. > > > > > > *Best Regards,* > > > *Zhenghua Gao* > > > > > > > > > On Tue, Apr 7, 2020 at 8:09 PM godfrey he wrote: > > > > > > > Hi Zhenghua, > > > > > > > > Thanks for driving this. It's one step forward that TableEnvironment > > > > supports more complete SQLs. > > > > I have a few minor questions: > > > > 1. regarding to "Interoperability between Flink and Hive is not > > > > guaranteed", can you explain this more? > > > > 2. regarding to "The Grammar", Calcite does not support "CREATE VIEW > [ IF > > > > NOT EXISTS ]", instead supports "CREATE [OR REPLACE]". "SHOW VIEWS" > and > > > > "DESCRIBE VIEW xx" are not supported now. Calcite does not support > > > describe > > > > an object through "DESC" . I think It's better this flip can support > > > "SHOW > > > > VIEWS" and "DESCRIBE VIEW xx". > > > > > > > > Best, > > > > Godfrey > > > > > > > > Zhenghua Gao 于2020年4月3日周五 下午3:04写道： > > > > > > > > > Hi community, > > > > > > > > > > It's a long time since we started the discussion of supporting > view in > > > > > FLINK SQL. > > > > > Flink also continues to move forward since then. > > > > > FLINK-10232 introduces the grammar and FLINK-12905 supports > CatalogView > > > > > in blink planner. > > > > > The missing link is validate the view definition and store the > > > > > original/expanded text in the catalog. > > > > > I have updated the design doc of FLIP-71, > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL > > > > > > > > > > Any comments and feedbacks are welcome and appreciated. Thanks. > > > > > > > > > > *Best Regards,* > > > > > *Zhenghua Gao* > > > > > > > > > > > > > > > On Tue, Sep 17, 2019 at 11:51 AM Zhenghua Gao > wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > In umbrella task FLINK-10232 we have introduced CREATE/DROP VIEW > > > grammar > > > > > > in our module flink-sql-parser. But we don't support view > objects in > > > > > > neither blink planner nor old planner. > > > > > > > > > > > > I'd like to kick off a discussion on end to end view support in > Flink > > > > > > SQL in blink planner. It's helpful to improve the usability of > the > > > > > > framework for SQL users. > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/14bx0t8wYH7_o4ChNkDoBFGn-i0T-Q7kUiOFvDd13_Fk/edit#heading=h.m031smarjj9p > > > > > > > > > > > > In short, it: > > > > > > - support define views and store them in catalog > > > > > > - support drop view definitions from catalog > > > > > > - support query views > > > > > > - support other view related DDLs > > > > > > > > > > > > Any comments and feedbacks are welcome and appreciated. Thanks. > > > > > > > > > > > > *Best Regards,* > > > > > > *Zhenghua Gao* > > > > > > > > > > > > > > >
Fwd: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
forward the reply to ML too. *Best Regards,* *Zhenghua Gao* -- Forwarded message - From: Zhenghua Gao Date: Tue, Apr 7, 2020 at 8:40 PM Subject: Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL To: godfrey he >> regarding to "Interoperability between Flink and Hive is not guaranteed", can you explain this more? We have several limitations of interoperability between flink objects and hive objects (tables, functions, etc). So we don't promise the interoperability of views between flink and hive since a view is defined base on these objects. >> "CREATE VIEW [ IF NOT EXISTS ]" This should be "CREATE VIEW [OR REPLACE]". >> "DESC" It's a shortcut of "DESCRIBE" in SQL Client (See desc table xxx). In DDL, we should only support "SHOW VIEWS" and "DESCRIBE VIEW xxx". I have updated the design doc, thanks. *Best Regards,* *Zhenghua Gao* On Tue, Apr 7, 2020 at 8:09 PM godfrey he wrote: > Hi Zhenghua, > > Thanks for driving this. It's one step forward that TableEnvironment > supports more complete SQLs. > I have a few minor questions: > 1. regarding to "Interoperability between Flink and Hive is not > guaranteed", can you explain this more? > 2. regarding to "The Grammar", Calcite does not support "CREATE VIEW [ IF > NOT EXISTS ]", instead supports "CREATE [OR REPLACE]". "SHOW VIEWS" and > "DESCRIBE VIEW xx" are not supported now. Calcite does not support describe > an object through "DESC" . I think It's better this flip can support "SHOW > VIEWS" and "DESCRIBE VIEW xx". > > Best, > Godfrey > > Zhenghua Gao 于2020年4月3日周五 下午3:04写道： > >> Hi community, >> >> It's a long time since we started the discussion of supporting view in >> FLINK SQL. >> Flink also continues to move forward since then. >> FLINK-10232 introduces the grammar and FLINK-12905 supports CatalogView >> in blink planner. >> The missing link is validate the view definition and store the >> original/expanded text in the catalog. >> I have updated the design doc of FLIP-71, >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL >> >> Any comments and feedbacks are welcome and appreciated. Thanks. >> >> *Best Regards,* >> *Zhenghua Gao* >> >> >> On Tue, Sep 17, 2019 at 11:51 AM Zhenghua Gao wrote: >> >>> Hi folks, >>> >>> In umbrella task FLINK-10232 we have introduced CREATE/DROP VIEW grammar >>> in our module flink-sql-parser. But we don't support view objects in >>> neither blink planner nor old planner. >>> >>> I'd like to kick off a discussion on end to end view support in Flink >>> SQL in blink planner. It's helpful to improve the usability of the >>> framework for SQL users. >>> >>> >>> https://docs.google.com/document/d/14bx0t8wYH7_o4ChNkDoBFGn-i0T-Q7kUiOFvDd13_Fk/edit#heading=h.m031smarjj9p >>> >>> In short, it: >>> - support define views and store them in catalog >>> - support drop view definitions from catalog >>> - support query views >>> - support other view related DDLs >>> >>> Any comments and feedbacks are welcome and appreciated. Thanks. >>> >>> *Best Regards,* >>> *Zhenghua Gao* >>> >>
Re: [DISCUSS] FLIP-71 - E2E View support in Flink SQL
Hi community, It's a long time since we started the discussion of supporting view in FLINK SQL. Flink also continues to move forward since then. FLINK-10232 introduces the grammar and FLINK-12905 supports CatalogView in blink planner. The missing link is validate the view definition and store the original/expanded text in the catalog. I have updated the design doc of FLIP-71, https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL Any comments and feedbacks are welcome and appreciated. Thanks. *Best Regards,* *Zhenghua Gao* On Tue, Sep 17, 2019 at 11:51 AM Zhenghua Gao wrote: > Hi folks, > > In umbrella task FLINK-10232 we have introduced CREATE/DROP VIEW grammar > in our module flink-sql-parser. But we don't support view objects in > neither blink planner nor old planner. > > I'd like to kick off a discussion on end to end view support in Flink SQL > in blink planner. It's helpful to improve the usability of the framework > for SQL users. > > > https://docs.google.com/document/d/14bx0t8wYH7_o4ChNkDoBFGn-i0T-Q7kUiOFvDd13_Fk/edit#heading=h.m031smarjj9p > > In short, it: > - support define views and store them in catalog > - support drop view definitions from catalog > - support query views > - support other view related DDLs > > Any comments and feedbacks are welcome and appreciated. Thanks. > > *Best Regards,* > *Zhenghua Gao* >
Re: [DISCUSS] Change default planner to blink planner in 1.11
+1 to make blink planner as the default. It's more powerful and has richer features. *Best Regards,* *Zhenghua Gao* On Thu, Apr 2, 2020 at 11:12 AM zoudan wrote: > > +1 to make the blink planner be the planner in 1.11. > We have already user blink planner as default planner in our production > environment, and it works well. > > Best, > Dan Zou > > > > 在 2020年4月1日，22:39，Danny Chan 写道： > > > > +1 to use blink-planner as the default. It’s really awful to upgrade > Calcite for both planners each time. I believe user would also want a > default planner that is more powerful. > > > > Best, > > Danny Chan > > 在 2020年4月1日 +0800 PM10:26，Leonard Xu ，写道： > >> +1 to make the Blink planner be the planner in 1.11. > >> > >> Blink planner has richer features than legacy planner and stable enough > from now. > >> > >> Best, > >> Leonard Xu > >> > >>> 在 2020年4月1日，19:51，Aljoscha Krettek 写道： > >>> > >>> +1 to making Blink the default planner, we definitely don't want to > maintain two planners for much longer. > >>> > >>> Best, > >>> Aljoscha > >> > >
Re: [DISCUSS] FLIP-122: New Connector Property Keys for New Factory
Hi Jark, Thanks for the proposal. I'm +1 since it's more simple and clear for sql users. I have a question about this: does this affect descriptors and related validators? *Best Regards,* *Zhenghua Gao* On Mon, Mar 30, 2020 at 2:02 PM Jark Wu wrote: > Hi everyone, > > I want to start a discussion about further improve and simplify our current > connector porperty keys, aka WITH options. Currently, we have a > 'connector.' prefix for many properties, but they are verbose, and we see a > big inconsistency between the properties when designing FLIP-107. > > So we propose to remove all the 'connector.' prefix and rename > 'connector.type' to 'connector', 'format.type' to 'format'. So a new Kafka > DDL may look like this: > > CREATE TABLE kafka_table ( > ... > ) WITH ( > 'connector' = 'kafka', > 'version' = '0.10', > 'topic' = 'test-topic', > 'startup-mode' = 'earliest-offset', > 'properties.bootstrap.servers' = 'localhost:9092', > 'properties.group.id' = 'testGroup', > 'format' = 'json', > 'format.fail-on-missing-field' = 'false' > ); > > The new connector property key set will come together with new Factory > inferface which is proposed in FLIP-95. Old properties are still compatible > with their existing implementation. New properties are only available in > new DynamicTableFactory implementations. > > You can access the detailed FLIP here: > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > > Best, > Jark >
[jira] [Created] (FLINK-16800) TypeMappingUtils#checkPhysicalLogicalTypeCompatible didn't deal with nested types
Zhenghua Gao created FLINK-16800:
Summary: TypeMappingUtils#checkPhysicalLogicalTypeCompatible
didn't deal with nested types
Key: FLINK-16800
URL: https://issues.apache.org/jira/browse/FLINK-16800
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.11.0
the planner will use TypeMappingUtils#checkPhysicalLogicalTypeCompatible to
validate logical schema and physical schema are compatible when translate
CatalogSinkModifyOperation to Calcite relational expression. The validation
didn't deal with nested types well, which could expose the following
ValidationException:
{code:java}
Exception in thread "main" org.apache.flink.table.api.ValidationException:
Type ARRAY> of table field 'old'
does not match with the physical type ARRAY> of the 'old' field of the TableSource return
type.
at
org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:164)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:277)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:254)
at
org.apache.flink.table.types.logical.utils.LogicalTypeDefaultVisitor.visit(LogicalTypeDefaultVisitor.java:157)
at org.apache.flink.table.types.logical.ArrayType.accept(ArrayType.java:110)
at
org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappingUtils.java:254)
at
org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:160)
at
org.apache.flink.table.utils.TypeMappingUtils.lambda$computeInCompositeType$8(TypeMappingUtils.java:232)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.flink.table.utils.TypeMappingUtils.computeInCompositeType(TypeMappingUtils.java:214)
at
org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndices(TypeMappingUtils.java:192)
at
org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndicesOrTimeAttributeMarkers(TypeMappingUtils.java:112)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.computeIndexMapping(StreamExecTableSourceScan.scala:212)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:107)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:62)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlan(StreamExecTableSourceScan.scala:62)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:84)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:44)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlan(StreamExecExchange.scala:44)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:161)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:51)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlan(StreamExecLimit.scala:51)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToTransformation(StreamExecSink.scala:184)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:153)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:48)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSi
[jira] [Created] (FLINK-16327) Add TableEnvironment.fromElements interfaces for usability
Zhenghua Gao created FLINK-16327:
Summary: Add TableEnvironment.fromElements interfaces for usability
Key: FLINK-16327
URL: https://issues.apache.org/jira/browse/FLINK-16327
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Zhenghua Gao
h1. Interface
{code:java}
/**
* Creates a table from a group of objects (known as its elements). The
schema of the table
* would be inferred from the type of elements.
*
* @param data a group of objects.
*/
Table fromElements(Collection data);
/**
* Creates a table from a group of objects (known as its elements). The
schema of the table
* would be inferred from the passed in data type.
*
* @param data a group of objects
* @param dataType the data type of the data
*/
Table fromElements(Collection data, DataType dataType);
{code}
h1. Use Case
* One potential use case for Table API
{code:java}
@Test
def testUnregisteredCollectionSource1(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
tEnv.fromElements(data.asJava)
.as('first, 'id, 'score, 'last)
.where('id > 4)
.select('last, 'score * 2)
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
@Test
def testUnregisteredCollectionSource2(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val dataType = DataTypes.ROW(
DataTypes.FIELD("first", DataTypes.STRING()),
DataTypes.FIELD("id", DataTypes.INT()),
DataTypes.FIELD("score", DataTypes.DOUBLE()),
DataTypes.FIELD("last", DataTypes.STRING()))
tEnv.fromElements(data.asJava, dataType)
.where('id > 4)
.select('last, 'score * 2)
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
{code}
* One potential use case for SQL
{code:java}
@Test
def testUnregisteredCollectionSource1(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val table = tEnv.fromElements(data.asJava).as('first, 'id, 'score, 'last)
tEnv.createTemporaryView("T", table)
tEnv.sqlQuery("SELECT last, score * 2 FROM T WHERE id > 4")
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
@Test
def testUnregisteredCollectionSource2(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val dataType = DataTypes.ROW(
DataTypes.FIELD("first", DataTypes.STRING()),
DataTypes.FIELD("id", DataTypes.INT()),
DataTypes.FIELD("score", DataTypes.DOUBLE()),
DataTypes.FIELD("last", DataTypes.STRING()))
val table = tEnv.fromElements(data.asJava, dataType)
tEnv.createTemporaryView("T", table)
tEnv.sqlQuery("SELECT last, score * 2 FROM T WHERE id > 4")
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
{code}
h1. The proposal
* data type inference
We need to infer the data type from the data for the first interface. A
potential tool is the DataTypeExtractor, but it doesn't support scala.tuple,
Row, etc. For the most popular in our test cases Row or scala.tuple type, we
could enumerate and use a recursive traversal method to get all available types
of underlying objects. This can solve most of the cases and improve usability.
* proposed changes
** A CollectionQueryOperation which implements QueryOperation to describe the
relational operation
** The logical and physical RelNode for legacy planner. In the physical node,
we can translate the data to DataStream
** The logical and physical RelNode for blink planner. In the physical node,
we can translate the data to Transformation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
Congrats Jingsong！ *Best Regards,* *Zhenghua Gao* On Fri, Feb 21, 2020 at 11:59 AM godfrey he wrote: > Congrats Jingsong! Well deserved. > > Best, > godfrey > > Jeff Zhang 于2020年2月21日周五 上午11:49写道： > >> Congratulations！Jingsong. You deserve it >> >> wenlong.lwl 于2020年2月21日周五 上午11:43写道： >> >>> Congrats Jingsong! >>> >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu wrote: >>> >>> > Congrats Jingsong! >>> > >>> > > 在 2020年2月21日，上午11:39，Jark Wu 写道： >>> > > >>> > > Congratulations Jingsong! Well deserved. >>> > > >>> > > Best, >>> > > Jark >>> > > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan wrote: >>> > > >>> > >> Congratulations! Jingsong >>> > >> >>> > >> >>> > >> Best, >>> > >> Dan Zou >>> > >> >>> > >>> > >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> >
[jira] [Created] (FLINK-16160) Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path
Zhenghua Gao created FLINK-16160: Summary: Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path Key: FLINK-16160 URL: https://issues.apache.org/jira/browse/FLINK-16160 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao In ConnectTableDescriptor#createTemporaryTable, the proctime/rowtime properties are ignored so the generated catalog table is not correct. We should fix this to let TableEnvironment#connect() support watermark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16117) Avoid register source in TableTestBase#addTableSource
Zhenghua Gao created FLINK-16117: Summary: Avoid register source in TableTestBase#addTableSource Key: FLINK-16117 URL: https://issues.apache.org/jira/browse/FLINK-16117 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao This affects thousands of unit tests: 1) explainSourceAsString of CatalogSourceTable changes 2)JoinTest#testUDFInJoinCondition: SQL keywords must be escaped 3) GroupWindowTest#testTimestampEventTimeTumblingGroupWindowWithProperties: Reference to a rowtime or proctime window required 4) SetOperatorsTest#testInWithProject: legacy type vs new type -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16029) Remove register source and sink in test cases of planner
Zhenghua Gao created FLINK-16029: Summary: Remove register source and sink in test cases of planner Key: FLINK-16029 URL: https://issues.apache.org/jira/browse/FLINK-16029 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao Many test cases of planner use TableEnvironement.registerTableSource() and registerTableSink() which should be avoid。We want to refactor these cases via TableEnvironment.connect(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15968) LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO
Zhenghua Gao created FLINK-15968: Summary: LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO Key: FLINK-15968 URL: https://issues.apache.org/jira/browse/FLINK-15968 Project: Flink Issue Type: Bug Components: Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.11.0 Reporter: Zhenghua Gao Currently LegacyTypeInfoDataTypeConverter only support conversion between DataTypes.BYTES and BYTE_PRIMITIVE_ARRAY_TYPE_INFO. When we update connectors to new type system, we need to convert BINARY(n) or VARBINARY(n) to BYTE_PRIMITIVE_ARRAY_TYPE_INFO. The Hive connector achieve this via depending blink planner‘s conversion logic, which is odd because a planner dependency won't be necessary for connectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Improve TableFactory to add Context
+1 (non-binding). Thanks for driving this. *Best Regards,* *Zhenghua Gao* On Fri, Feb 7, 2020 at 5:05 PM Leonard Xu wrote: > +1(non-binding), nice design! > after read full discussion mail list. > > Best, > Leonard Xu > > > 在 2020年2月6日，23:12，Timo Walther 写道： > > > > +1 > > > > On 06.02.20 05:54, Bowen Li wrote: > >> +1, LGTM > >> On Tue, Feb 4, 2020 at 11:28 PM Jark Wu wrote: > >>> +1 form my side. > >>> Thanks for driving this. > >>> > >>> Btw, could you also attach a JIRA issue with the changes described in > it, > >>> so that users can find the issue through the mailing list in the > future. > >>> > >>> Best, > >>> Jark > >>> > >>> On Wed, 5 Feb 2020 at 13:38, Kurt Young wrote: > >>> > >>>> +1 from my side. > >>>> > >>>> Best, > >>>> Kurt > >>>> > >>>> > >>>> On Wed, Feb 5, 2020 at 10:59 AM Jingsong Li > >>>> wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> Interface updated. > >>>>> Please re-vote. > >>>>> > >>>>> Best, > >>>>> Jingsong Lee > >>>>> > >>>>> On Tue, Feb 4, 2020 at 1:28 PM Jingsong Li > >>>> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> I would like to start the vote for the improve of > >>>>>> TableFactory, which is discussed and > >>>>>> reached a consensus in the discussion thread[2]. > >>>>>> > >>>>>> The vote will be open for at least 72 hours. I'll try to close it > >>>>>> unless there is an objection or not enough votes. > >>>>>> > >>>>>> [1] > >>>>>> > >>>>> > >>>> > >>> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improve-TableFactory-td36647.html > >>>>>> > >>>>>> Best, > >>>>>> Jingsong Lee > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best, Jingsong Lee > >>>>> > >>>> > >>> > > > >
Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system
Thanks Timo! Look forward your design! *Best Regards,* *Zhenghua Gao* On Fri, Feb 7, 2020 at 5:26 PM Timo Walther wrote: > Hi Zhenghua, > > Jark is right. The reason why we haven't updated those interfaces yet is > because we are actually would like to introduce new interfaces. We > should target new interfaces in this release. Even a short-term fix as > you proposed with `getRecordDataType` does actually not help as Jingsong > pointed out because we cannot represent tuples in DataType and are also > not planning to support them natively but only as a structured type in > the future. > > In my envisioned design, the new sink interface should just always get a > `ChangeRow` which is never serialized and just a data structure for > communicating between the wrapping sink function and the returned sink > function by the table sink. > > Let me sketch a rough design document that I will share with you > shortly. Then we could also discuss alternatives. > > Thanks, > Timo > > > On 04.02.20 04:18, Zhenghua Gao wrote: > > Hi Jark, thanks for your comments. > >>>> IIUC, the framework will only recognize getRecordDataType and > >>>> ignore getConsumedDataType for UpsertStreamTableSink, is that right? > > Your are right. > > > >>>> getRecordDataType is little confused as UpsertStreamTableSink already > has > >>>> three getXXXType(). > > the getRecordType and getOutputType is deprecated and mainly for backward > > compatibility. > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Mon, Feb 3, 2020 at 10:11 PM Jark Wu wrote: > > > >> Thanks Zhenghua for starting this discussion. > >> > >> Currently, all the UpsertStreamTableSinks can't upgrade to the new type > >> system which affects usability a lot. > >> I hope we can fix that in 1.11. > >> > >> I'm find with *getRecordDataType* for a temporary solution. > >> IIUC, the framework will only recognize getRecordDataType and > >> ignore getConsumedDataType for UpsertStreamTableSink, is that right? > >> > >> I guess Timo are planning to design a new source/sink interface which > will > >> also fix this problem, but I'm not sure the timelines. cc @Timo > >> It would be better if we can have a new and complete interface, because > >> getRecordDataType is little confused as UpsertStreamTableSink already > has > >> three getXXXType(). > >> > >> Best, > >> Jark > >> > >> > >> On Mon, 3 Feb 2020 at 17:48, Zhenghua Gao wrote: > >> > >>> Hi Jingsong, For now, only UpsertStreamTableSink and > >>> RetractStreamTableSink consumes JTuple2 > >>> So the 'getConsumedDataType' interface is not necessary in validate & > >>> codegen phase. > >>> See > >>> > >>> > >> > https://github.com/apache/flink/pull/10880/files#diff-137d090bd719f3a99ae8ba6603ed81ebR52 > >>> and > >>> > >>> > >> > https://github.com/apache/flink/pull/10880/files#diff-8405c17e5155aa63d16e497c4c96a842R304 > >>> > >>> What about stay the same to use RAW type? > >>> > >>> *Best Regards,* > >>> *Zhenghua Gao* > >>> > >>> > >>> On Mon, Feb 3, 2020 at 4:59 PM Jingsong Li > >> wrote: > >>> > >>>> Hi Zhenghua, > >>>> > >>>> The *getRecordDataType* looks good to me. > >>>> > >>>> But the main problem is how to represent the tuple type in DataType. I > >>>> understand that it is necessary to use StructuredType, but at present, > >>>> planner does not support StructuredType, so the other way is to > support > >>>> StructuredType. > >>>> > >>>> Best, > >>>> Jingsong Lee > >>>> > >>>> On Mon, Feb 3, 2020 at 4:49 PM Kurt Young wrote: > >>>> > >>>>> Would overriding `getConsumedDataType` do the job? > >>>>> > >>>>> Best, > >>>>> Kurt > >>>>> > >>>>> > >>>>> On Mon, Feb 3, 2020 at 3:52 PM Zhenghua Gao > >> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> FLINK-12254[1] [2] updated TableSink and related interfaces to new > >>> type > >>>>>> system which > >>>>>> allows connectors use the new type system based on DataTypes. > >>>>>> > >>>>>> But FLINK-12911 port UpsertStreamTableSink and > >> RetractStreamTableSink > >>> to > >>>>>> flink-api-java-bridge and returns TypeInformation of the requested > >>>> record > >>>>>> type which > >>>>>> can't support types with precision and scale, e.g. TIMESTAMP(p), > >>>>>> DECIMAL(p,s). > >>>>>> > >>>>>> /** > >>>>>> * Returns the requested record type. > >>>>>> */ > >>>>>> TypeInformation getRecordType(); > >>>>>> > >>>>>> > >>>>>> A proposal is deprecating the *getRecordType* API and adding a > >>>>>> *getRecordDataType* API instead to return the data type of the > >>> requested > >>>>>> record. I have filed the issue FLINK-15469 and > >>>>>> an initial PR to verify it. > >>>>>> > >>>>>> What do you think about this API changes? Any feedback are > >>> appreciated. > >>>>>> [1] https://issues.apache.org/jira/browse/FLINK-12254 > >>>>>> [2] https://github.com/apache/flink/pull/8596 > >>>>>> [3] https://issues.apache.org/jira/browse/FLINK-15469 > >>>>>> > >>>>>> *Best Regards,* > >>>>>> *Zhenghua Gao* > >>>>>> > >>>>> > >>>> > >>>> -- > >>>> Best, Jingsong Lee > >>>> > >>> > >> > > > >
Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor
+1 to remove these methods. One concern about invocations of TableSource::getTableSchema: By removing such methods, we can stop calling TableSource::getTableSchema in some place(such as BatchTableEnvImpl/TableEnvironmentImpl#validateTableSource, ConnectorCatalogTable, TableSourceQueryOperation). But in other place we need field types and names of the table source(such as BatchExecLookupJoinRule/StreamExecLookupJoinRule, PushProjectIntoTableSourceScanRule, CommonLookupJoin). So how should we deal with this？ *Best Regards,* *Zhenghua Gao* On Wed, Feb 5, 2020 at 2:36 PM Kurt Young wrote: > Hi all, > > I'd like to bring up a discussion about removing registration of > TableSource and > TableSink in TableEnvironment as well as in ConnectTableDescriptor. The > affected > method would be: > > TableEnvironment::registerTableSource > TableEnvironment::fromTableSource > TableEnvironment::registerTableSink > ConnectTableDescriptor::registerTableSource > ConnectTableDescriptor::registerTableSink > ConnectTableDescriptor::registerTableSourceAndSink > > (Most of them are already deprecated, except for > TableEnvironment::fromTableSource, > which was intended to deprecate but missed by accident). > > FLIP-64 [1] already explained why we want to deprecate TableSource & > TableSink from > user's interface. In a short word, these interfaces should only read & > write the physical > representation of the table, and they are not fitting well after we already > introduced some > logical table fields such as computed column, watermarks. > > Another reason is the exposure of registerTableSource in Table Env just > make the whole > SQL protocol opposite. TableSource should be used as a reader of table, it > should rely on > other metadata information held by framework, which eventually comes from > DDL or > ConnectDescriptor. But if we register a TableSource to Table Env, we have > no choice but > have to rely on TableSource::getTableSchema. It will make the design > obscure, sometimes > TableSource should trust the information comes from framework, but > sometimes it should > also generate its own schema information. > > Furthermore, if the authority about schema information is not clear, it > will make things much > more complicated if we want to improve the table api usability such as > introducing automatic > schema inference in the near future. > > Since this is an API break change, I've also included user mailing list to > gather more feedbacks. > > Best, > Kurt > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module >
Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system
Hi Jark, thanks for your comments. >>>IIUC, the framework will only recognize getRecordDataType and >>>ignore getConsumedDataType for UpsertStreamTableSink, is that right? Your are right. >>>getRecordDataType is little confused as UpsertStreamTableSink already has >>>three getXXXType(). the getRecordType and getOutputType is deprecated and mainly for backward compatibility. *Best Regards,* *Zhenghua Gao* On Mon, Feb 3, 2020 at 10:11 PM Jark Wu wrote: > Thanks Zhenghua for starting this discussion. > > Currently, all the UpsertStreamTableSinks can't upgrade to the new type > system which affects usability a lot. > I hope we can fix that in 1.11. > > I'm find with *getRecordDataType* for a temporary solution. > IIUC, the framework will only recognize getRecordDataType and > ignore getConsumedDataType for UpsertStreamTableSink, is that right? > > I guess Timo are planning to design a new source/sink interface which will > also fix this problem, but I'm not sure the timelines. cc @Timo > It would be better if we can have a new and complete interface, because > getRecordDataType is little confused as UpsertStreamTableSink already has > three getXXXType(). > > Best, > Jark > > > On Mon, 3 Feb 2020 at 17:48, Zhenghua Gao wrote: > > > Hi Jingsong, For now, only UpsertStreamTableSink and > > RetractStreamTableSink consumes JTuple2 > > So the 'getConsumedDataType' interface is not necessary in validate & > > codegen phase. > > See > > > > > https://github.com/apache/flink/pull/10880/files#diff-137d090bd719f3a99ae8ba6603ed81ebR52 > > and > > > > > https://github.com/apache/flink/pull/10880/files#diff-8405c17e5155aa63d16e497c4c96a842R304 > > > > What about stay the same to use RAW type? > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Mon, Feb 3, 2020 at 4:59 PM Jingsong Li > wrote: > > > > > Hi Zhenghua, > > > > > > The *getRecordDataType* looks good to me. > > > > > > But the main problem is how to represent the tuple type in DataType. I > > > understand that it is necessary to use StructuredType, but at present, > > > planner does not support StructuredType, so the other way is to support > > > StructuredType. > > > > > > Best, > > > Jingsong Lee > > > > > > On Mon, Feb 3, 2020 at 4:49 PM Kurt Young wrote: > > > > > > > Would overriding `getConsumedDataType` do the job? > > > > > > > > Best, > > > > Kurt > > > > > > > > > > > > On Mon, Feb 3, 2020 at 3:52 PM Zhenghua Gao > wrote: > > > > > > > >> Hi all, > > > >> > > > >> FLINK-12254[1] [2] updated TableSink and related interfaces to new > > type > > > >> system which > > > >> allows connectors use the new type system based on DataTypes. > > > >> > > > >> But FLINK-12911 port UpsertStreamTableSink and > RetractStreamTableSink > > to > > > >> flink-api-java-bridge and returns TypeInformation of the requested > > > record > > > >> type which > > > >> can't support types with precision and scale, e.g. TIMESTAMP(p), > > > >> DECIMAL(p,s). > > > >> > > > >> /** > > > >> * Returns the requested record type. > > > >> */ > > > >> TypeInformation getRecordType(); > > > >> > > > >> > > > >> A proposal is deprecating the *getRecordType* API and adding a > > > >> *getRecordDataType* API instead to return the data type of the > > requested > > > >> record. I have filed the issue FLINK-15469 and > > > >> an initial PR to verify it. > > > >> > > > >> What do you think about this API changes? Any feedback are > > appreciated. > > > >> [1] https://issues.apache.org/jira/browse/FLINK-12254 > > > >> [2] https://github.com/apache/flink/pull/8596 > > > >> [3] https://issues.apache.org/jira/browse/FLINK-15469 > > > >> > > > >> *Best Regards,* > > > >> *Zhenghua Gao* > > > >> > > > > > > > > > > -- > > > Best, Jingsong Lee > > > > > >
Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system
Should we distinguish *record data type* and *consumed data type*? Currently the design of UpsertStreamTableSink and RetractStreamTableSink DO distinguish them. In my proposal the framework will ignore *getConsumedDataType*, so it's ok to use *getConsumedDataType* to do the job if we don't distinguish *record data type* and *consumed data type*. *Best Regards,* *Zhenghua Gao* On Mon, Feb 3, 2020 at 4:49 PM Kurt Young wrote: > Would overriding `getConsumedDataType` do the job? > > Best, > Kurt > > > On Mon, Feb 3, 2020 at 3:52 PM Zhenghua Gao wrote: > > > Hi all, > > > > FLINK-12254[1] [2] updated TableSink and related interfaces to new type > > system which > > allows connectors use the new type system based on DataTypes. > > > > But FLINK-12911 port UpsertStreamTableSink and RetractStreamTableSink to > > flink-api-java-bridge and returns TypeInformation of the requested record > > type which > > can't support types with precision and scale, e.g. TIMESTAMP(p), > > DECIMAL(p,s). > > > > /** > > * Returns the requested record type. > > */ > > TypeInformation getRecordType(); > > > > > > A proposal is deprecating the *getRecordType* API and adding a > > *getRecordDataType* API instead to return the data type of the requested > > record. I have filed the issue FLINK-15469 and > > an initial PR to verify it. > > > > What do you think about this API changes? Any feedback are appreciated. > > [1] https://issues.apache.org/jira/browse/FLINK-12254 > > [2] https://github.com/apache/flink/pull/8596 > > [3] https://issues.apache.org/jira/browse/FLINK-15469 > > > > *Best Regards,* > > *Zhenghua Gao* > > >
Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system
Hi Jingsong, For now, only UpsertStreamTableSink and RetractStreamTableSink consumes JTuple2 So the 'getConsumedDataType' interface is not necessary in validate & codegen phase. See https://github.com/apache/flink/pull/10880/files#diff-137d090bd719f3a99ae8ba6603ed81ebR52 and https://github.com/apache/flink/pull/10880/files#diff-8405c17e5155aa63d16e497c4c96a842R304 What about stay the same to use RAW type? *Best Regards,* *Zhenghua Gao* On Mon, Feb 3, 2020 at 4:59 PM Jingsong Li wrote: > Hi Zhenghua, > > The *getRecordDataType* looks good to me. > > But the main problem is how to represent the tuple type in DataType. I > understand that it is necessary to use StructuredType, but at present, > planner does not support StructuredType, so the other way is to support > StructuredType. > > Best, > Jingsong Lee > > On Mon, Feb 3, 2020 at 4:49 PM Kurt Young wrote: > > > Would overriding `getConsumedDataType` do the job? > > > > Best, > > Kurt > > > > > > On Mon, Feb 3, 2020 at 3:52 PM Zhenghua Gao wrote: > > > >> Hi all, > >> > >> FLINK-12254[1] [2] updated TableSink and related interfaces to new type > >> system which > >> allows connectors use the new type system based on DataTypes. > >> > >> But FLINK-12911 port UpsertStreamTableSink and RetractStreamTableSink to > >> flink-api-java-bridge and returns TypeInformation of the requested > record > >> type which > >> can't support types with precision and scale, e.g. TIMESTAMP(p), > >> DECIMAL(p,s). > >> > >> /** > >> * Returns the requested record type. > >> */ > >> TypeInformation getRecordType(); > >> > >> > >> A proposal is deprecating the *getRecordType* API and adding a > >> *getRecordDataType* API instead to return the data type of the requested > >> record. I have filed the issue FLINK-15469 and > >> an initial PR to verify it. > >> > >> What do you think about this API changes? Any feedback are appreciated. > >> [1] https://issues.apache.org/jira/browse/FLINK-12254 > >> [2] https://github.com/apache/flink/pull/8596 > >> [3] https://issues.apache.org/jira/browse/FLINK-15469 > >> > >> *Best Regards,* > >> *Zhenghua Gao* > >> > > > > -- > Best, Jingsong Lee >
[DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system
Hi all, FLINK-12254[1] [2] updated TableSink and related interfaces to new type system which allows connectors use the new type system based on DataTypes. But FLINK-12911 port UpsertStreamTableSink and RetractStreamTableSink to flink-api-java-bridge and returns TypeInformation of the requested record type which can't support types with precision and scale, e.g. TIMESTAMP(p), DECIMAL(p,s). /** * Returns the requested record type. */ TypeInformation getRecordType(); A proposal is deprecating the *getRecordType* API and adding a *getRecordDataType* API instead to return the data type of the requested record. I have filed the issue FLINK-15469 and an initial PR to verify it. What do you think about this API changes? Any feedback are appreciated. [1] https://issues.apache.org/jira/browse/FLINK-12254 [2] https://github.com/apache/flink/pull/8596 [3] https://issues.apache.org/jira/browse/FLINK-15469 *Best Regards,* *Zhenghua Gao*
Re: [VOTE] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL
+1 (non-binding) *Best Regards,* *Zhenghua Gao* On Wed, Jan 15, 2020 at 10:11 AM Danny Chan wrote: > +1 (non-binding) > > Best, > Danny Chan > 在 2019年12月31日 +0800 PM5:09，Forward Xu ，写道： > > Hi all, > > > > I'd like to start the vote of FLIP-90 [1] since that we have reached an > > agreement on the design in the discussion thread [2]. > > > > This vote will be open for at least 72 hours. Unless there is an > objection, > > I will try to close it by January 3, 2020 08:00 UTC if we have received > > sufficient votes. > > > > Best, > > ForwardXu > > > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550 > > [2] > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-JSON-functions-in-Flink-SQL-td32674.html >
Re: Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink
+1 (non-binding). Thanks for driving this. It's an important improvement in the discussed scenarios. *Best Regards,* *Zhenghua Gao* On Mon, Jan 13, 2020 at 6:13 PM Haibo Sun wrote: > +1 (non-binding) > > > Best, > Haibo > > At 2020-01-13 11:36:12, "Yun Gao" wrote: > >+1 (non-binding). > > > >Very thanks for introducing this topic back, and it should be able to > bring improvements in the discussed scenarios. > > > >Best, > >Yun > > > > > >-- > >From:Arvid Heise > >Send Time:2020 Jan. 10 (Fri.) 16:48 > >To:dev ; Zhijiang > >Subject:Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink > > > >non-binding +1 > > > >On Fri, Jan 10, 2020 at 9:11 AM Zhijiang .invalid> > >wrote: > > > >> +1, it is really nice to have the N-Ary stream operator which is > >> meaningful in some scenarios. > >> > >> best, > >> Zhijiang > >> > >> > >> -- > >> From:Jingsong Li > >> Send Time:2020 Jan. 10 (Fri.) 11:00 > >> To:dev > >> Subject:Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink > >> > >> +1 non-binding to the N-Ary Stream Operator. Thanks Piotr for driving. > >> Looks like the previous FLIP-92 did not change the "Next FLIP Number" in > >> FLIP page. > >> > >> Best, > >> Jingsong Lee > >> > >> On Fri, Jan 10, 2020 at 8:40 AM Benchao Li wrote: > >> > >> > Hi Piotr, > >> > > >> > It seems that we have the 'FLIP-92' already. > >> > see: > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+JDBC+catalog+and+Postgres+catalog > >> > > >> > > >> > Piotr Nowojski 于2020年1月9日周四 下午11:25写道： > >> > > >> > > Hi, > >> > > > >> > > I would like to start a vote for adding the N-Ary Stream Operator in > >> > Flink > >> > > as discussed in the discussion thread [1]. > >> > > > >> > > This vote will be opened at least until Wednesday, January 15th 8:00 > >> UTC. > >> > > > >> > > Piotrek > >> > > > >> > > [1] > >> > > > >> > > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-N-Ary-Stream-Operator-td11341.html > >> > > < > >> > > > >> > > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-N-Ary-Stream-Operator-td11341.html > >> > > > > >> > > >> > > >> > > >> > -- > >> > > >> > Benchao Li > >> > School of Electronics Engineering and Computer Science, Peking > University > >> > Tel:+86-15650713730 > >> > Email: libenc...@gmail.com; libenc...@pku.edu.cn > >> > > >> > >> > >> -- > >> Best, Jingsong Lee > >> > >> >
Re: [DISCUSS] A mechanism to validate the precision of columns for connectors
Hi Jingsong Lee You are right that the connectors don't validate data types either now. We seems lack a mechanism to validate with properties[1], data types, etc for CREATE TABLE. [1] https://issues.apache.org/jira/browse/FLINK-15509 *Best Regards,* *Zhenghua Gao* On Fri, Jan 10, 2020 at 2:59 PM Jingsong Li wrote: > Hi Zhenghua, > > I think it's not just about precision of type. Connectors not validate the > types either. > Now there is "SchemaValidator", this validator is just used to validate > type properties. But not for connector type support. > I think we can have something like "DataTypeValidator" to help connectors > validating their type support. > > Consider current validator design, validator is called by connector itself. > it's more like a util class than a mechanism. > > Best, > Jingsong Lee > > On Fri, Jan 10, 2020 at 11:47 AM Zhenghua Gao wrote: > > > Hi dev, > > > > I'd like to kick off a discussion on a mechanism to validate the > precision > > of columns for some connectors. > > > > We come to an agreement that the user should be informed if the connector > > does not support the desired precision. And from the connector > developer's > > view, there are 3-levels information to be considered: > > > >- the ability of external systems (e.g. Apache Derby support > >TIMESTAMP(9), Mysql support TIMESTAMP(6), etc) > > > > Connector developers should use this information to validate user's DDL > and > > make sure throw an exception if concrete column is out of range. > > > > > >- schema of referenced tables in external systems > > > > If the schema information of referenced tables is available in Compile > > Time, connector developers could use it to find the mismatch between DDL. > > But in most cases, the schema information is unavailable because of > network > > isolation or authority management. We should use it with caution. > > > > > >- schema-less external systems (e.g. HBase) > > > > If the external systems is schema-less like HBase, the connector > developer > > should make sure the connector doesn't cause precision loss (e.g. > > flink-hbase serializes java.sql.Timestamp to long in bytes which only > keep > > millisecond's precision.) > > > > To make it more specific, some scenarios of JDBC Connector are list as > > following: > > > >- The underlying DB supports DECIMAL(65, 30), which is out of the > range > >of Flink's Decimal > >- The underlying DB supports TIMESTAMP(6), and user want to define a > >table with TIMESTAMP(9) in Flink > >- User defines a table with DECIMAL(10, 4) in underlying DB, and want > to > >define a table with DECIMAL(5, 2) in Flink > >- The precision of the underlying DB varies between different versions > > > > > > What do you think about this? any feedback are appreciates. > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > -- > Best, Jingsong Lee >
Re: [DISCUSS] FLIP-92: JDBC catalog and Postgres catalog
Hi Bowen, Thanks for driving this. I think it would be very convenience to use tables in external DBs with JDBC Catalog. I have one concern about "Flink-Postgres Data Type Mapping" part: In Postgress, the TIME/TIMESTAMP WITH TIME ZONE has the java.time.Instant semantic, and should be mapped to Flink's TIME/TIMESTAMP WITH LOCAL TIME ZONE *Best Regards,* *Zhenghua Gao* On Fri, Jan 10, 2020 at 11:09 AM Jingsong Li wrote: > Hi Bowen, thanks for reply and updating. > > > I don't see much value in providing a builder for jdbc catalogs, as they > only have 4 or 5 required params, no optional ones. I prefer users just > provide a base url without default db, usrname, pwd so we don't need to > parse url all around, as I mentioned jdbc catalog may need to establish > connections to different databases in a db instance, > > I suggest that the parameters can be completely consistent with the > JDBCTableSource / JDBCTableSink. > If you take a look to JDBC api: "DriverManager.getConnection". > That allow "default db, username, pwd" things optional. They can included > in URL. Of course JDBC api also allows establishing connections to > different databases in a db instance. > So I think we don't need provide a "base_url", we can just provide a real > "url". > To be consistent with JDBC api. > > Best, > Jingsong Lee > > On Fri, Jan 10, 2020 at 10:34 AM Jark Wu wrote: > > > Thanks Bowen for the reply, > > > > A user-facing JDBCCatalog and 'catalog.type' = 'jdbc' sounds good to me. > > > > I have some other minor comments when I went through the updated > > documentation: > > > > 1) 'base_url' configuration: We are following the configuration format > > guideline [1] which suggest to use dash (-) instead of underline (_). > > And I'm a little confused the meaning of "base_url" at the first > > glance, another idea is split it into several configurations: 'driver', > > 'hostname', 'port'. > > > > 2) 'default-database' is optional, then which database will be used or > what > > is the behavior when the default database is not selected. > > > > 3) a builder for jdbc catalogs: I agree with Jingsong to provide a > builder. > > Because there is optional configuration here (the default database), > >and providind Builder as the API will be easier for evolution, I'm not > > sure we won't add/modify parameters in the future. > > > > [1]: > > > > > https://flink.apache.org/contributing/code-style-and-quality-components.html#configuration-changes > > > > On Fri, 10 Jan 2020 at 04:52, Bowen Li wrote: > > > > > Hi Jark and Jingsong, > > > > > > Thanks for your review. Please see my reply in line. > > > > > > > why introducing a `PostgresJDBCCatalog`, not a generic `JDBCCatalog` > > > (catalog.type = 'postgres' vs 'jdbc') ? > > > > > > Thanks for the reminding and I looked at JDBCDialect. A generic, > > > user-facing JDBCCatalog with catalog.type = jdbc and find specific db > > > implementations (pg v.s. mysql v.s. ...) is more aligned with how jdbc > > > sink/source is handled, indeed. However, the catalogs would also need > to > > > execute the query and parse query results in a db-dependent way. E.g. > > jdbc > > > catalog needs to establish connections to different databases within a > db > > > instance on demand. So just having JDBCDialect won't be enough. > > > > > > I think we can do the following: > > > - provide a user-facing JDBCCatalog, composing a db-specific impl > like > > > PostgresJDBCCatalog and MySQLJDBCCatalog. Users still specify "jdbc" as > > > type in both Table API and SQL CLI, internally it will create a > > db-specific > > > impl depending on jdbc base url. > > > - some statements can reside in JDBCDialect. Query execution and > result > > > parsing logic would be located in db-specific impls. > > > > > > - We can provide a Builder for Catalog, In my opinion, defaultDatabase, > > > username, pwd can be included in JDBC DB url. > > > > > > I don't see much value in providing a builder for jdbc catalogs, as > they > > > only have 4 or 5 required params, no optional ones. I prefer users just > > > provide a base url without default db, usrname, pwd so we don't need to > > > parse url all around, as I mentioned jdbc catalog may need to
Re: [VOTE] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL
+1 from my side and thanks for driving this. *Best Regards,* *Zhenghua Gao* On Fri, Jan 10, 2020 at 11:10 AM Forward Xu wrote: > Hi Danny, > Thank you very much. > > Best, > Forward > > Danny Chan 于2020年1月10日周五 上午11:08写道： > > > Thanks Forward ~ > > +1 from my side and would review your Calcite PR this weekend :) Overall > > it looks good, and I believe we can merge it soon ~ > > > > Best, > > Danny Chan > > 在 2020年1月10日 +0800 AM11:04，Jark Wu ，写道： > > > Thanks Forward for driving this, > > > > > > The design doc looks very good to me. > > > +1 from my side. > > > > > > Best, > > > Jark > > > > > > On Thu, 9 Jan 2020 at 20:12, Forward Xu > wrote: > > > > > > > Hi all, > > > > > > > > Listened to the opinion of Timo since the last discussion and updated > > the > > > > document [1] Optimized the passing parameters of JSON table API. > Added > > > > return type when describing each JSON function. It makes the > > documentation > > > > more clear. So I again vote of FLIP-90 [2] since that we have reached > > an > > > > agreement on the design in the discussion thread [3]. > > > > This vote will be open for at least 72 hours. Unless there is an > > objection, > > > > I will try to close it by January 12, 2020 08:00 UTC if we have > > received > > > > sufficient votes. > > > > Best, > > > > ForwardXu > > > > > > > > [1] > > > > > > > > > > > https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp > > > > [2] > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550 > > > > [3] > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-JSON-functions-in-Flink-SQL-td32674.html > > > > > > >
[DISCUSS] A mechanism to validate the precision of columns for connectors
Hi dev, I'd like to kick off a discussion on a mechanism to validate the precision of columns for some connectors. We come to an agreement that the user should be informed if the connector does not support the desired precision. And from the connector developer's view, there are 3-levels information to be considered: - the ability of external systems (e.g. Apache Derby support TIMESTAMP(9), Mysql support TIMESTAMP(6), etc) Connector developers should use this information to validate user's DDL and make sure throw an exception if concrete column is out of range. - schema of referenced tables in external systems If the schema information of referenced tables is available in Compile Time, connector developers could use it to find the mismatch between DDL. But in most cases, the schema information is unavailable because of network isolation or authority management. We should use it with caution. - schema-less external systems (e.g. HBase) If the external systems is schema-less like HBase, the connector developer should make sure the connector doesn't cause precision loss (e.g. flink-hbase serializes java.sql.Timestamp to long in bytes which only keep millisecond's precision.) To make it more specific, some scenarios of JDBC Connector are list as following: - The underlying DB supports DECIMAL(65, 30), which is out of the range of Flink's Decimal - The underlying DB supports TIMESTAMP(6), and user want to define a table with TIMESTAMP(9) in Flink - User defines a table with DECIMAL(10, 4) in underlying DB, and want to define a table with DECIMAL(5, 2) in Flink - The precision of the underlying DB varies between different versions What do you think about this? any feedback are appreciates. *Best Regards,* *Zhenghua Gao*
[jira] [Created] (FLINK-15525) HBase connector should use new type system to suppport precision/scale
Zhenghua Gao created FLINK-15525: Summary: HBase connector should use new type system to suppport precision/scale Key: FLINK-15525 URL: https://issues.apache.org/jira/browse/FLINK-15525 Project: Flink Issue Type: Bug Components: Connectors / HBase Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Currently *HBaseTableSchema* use TypeInformation to specify an HBase table's schema, which would cause precision/scale loss for several data types. Meanwhile *HBaseTypeUtils* serialize a java.sql.Timestamp to long in bytes, which would cause precision loss for TIMESTAMP types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Rename terminology "Time-windowed Join" to "Interval Join" in Table API & SQL
+1 to align the terminology. *Best Regards,* *Zhenghua Gao* On Fri, Jan 3, 2020 at 12:59 PM Jingsong Li wrote: > +1 for this documentation change. > Hope less confuse to users. > > Best, > Jingsong Lee > > On Fri, Jan 3, 2020 at 12:09 PM Benchao Li wrote: > > > +1 > > > > It's good to align the terminology between Table API & SQL and > DataStream. > > > > Jark Wu 于2020年1月3日周五 下午12:04写道： > > > > > Hi everyone, > > > > > > As we discussed in the mailing list[1], the current "Time-windowed > Join" > > in > > > Table API & SQL is a little misleading which is not the same to "Window > > > Join" in DataStream, but the same to "Interval Join" in DataStream. > > > > > > So I would like to start a vote to rename the terminology of > > "Time-windowed > > > Join" to "Interval Join" in Table API & SQL **before 1.10 release**. > > > > > > Note that this is a purely documentation change, no updates for public > > API > > > or Javadocs. Updates for implementation codes (e.g. rename > > > DataStreamWindowJoin) is not targeted to 1.10. > > > > > > This vote will be open for at least 72 hours. Unless there is an > > objection. > > > This vote is required Consensus Approval which is the same to a FLIP > > vote. > > > > > > Best, > > > Jark > > > > > > [1]: > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Correct-the-terminology-of-quot-Time-windowed-Join-quot-to-quot-Interval-Join-quot-in-Table-L-td36202.html > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: libenc...@gmail.com; libenc...@pku.edu.cn > > > > > -- > Best, Jingsong Lee >
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
+1 for making blink planner as the default planner for SQL Client since we have made a huge improvement in 1.10. *Best Regards,* *Zhenghua Gao* On Sun, Jan 5, 2020 at 2:42 PM Benchao Li wrote: > +1 > > We have used blink planner since 1.9.0 release in our production > environment, and it behaves really impressive. > > Hequn Cheng 于2020年1月5日周日 下午1:58写道： > >> +1 to make blink planner as the default planner for SQL Client, hence we >> can give the blink planner a bit more exposure. >> >> Best, Hequn >> >> On Fri, Jan 3, 2020 at 6:32 PM Jark Wu wrote: >> >>> Hi Benoît, >>> >>> Thanks for the reminder. I will look into the issue and hopefully we can >>> target it into 1.9.2 and 1.10. >>> >>> Cheers, >>> Jark >>> >>> On Fri, 3 Jan 2020 at 18:21, Benoît Paris < >>> benoit.pa...@centraliens-lille.org> wrote: >>> >>>> > If anyone finds that blink planner has any significant defects and >>>> has a larger regression than the old planner, please let us know. >>>> >>>> Overall, the Blink-exclusive features are must (TopN, deduplicate, >>>> LAST_VALUE, plan reuse, etc)! But all use cases of the Legacy planner in >>>> production are not covered: >>>> An edge case of Temporal Table Functions does not allow computed Tables >>>> (as opposed to TableSources) to be used on the query side in Blink ( >>>> https://issues.apache.org/jira/browse/FLINK-14200) >>>> >>>> Cheers >>>> Ben >>>> >>>> >>>> On Fri, Jan 3, 2020 at 10:00 AM Jeff Zhang wrote: >>>> >>>>> +1, I have already made blink as the default planner of flink >>>>> interpreter in Zeppelin >>>>> >>>>> >>>>> Jingsong Li 于2020年1月3日周五 下午4:37写道： >>>>> >>>>>> Hi Jark, >>>>>> >>>>>> +1 for default blink planner in SQL-CLI. >>>>>> I believe this new planner can be put into practice in production. >>>>>> We've worked hard for nearly a year, but the old planner didn't move >>>>>> on. >>>>>> >>>>>> And I'd like to cc to u...@flink.apache.org. >>>>>> If anyone finds that blink planner has any significant defects and >>>>>> has a larger regression than the old planner, please let us know. We will >>>>>> be very grateful. >>>>>> >>>>>> Best, >>>>>> Jingsong Lee >>>>>> >>>>>> On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: >>>>>> >>>>>>> +1 for this. >>>>>>> We bring many SQL/API features and enhance stability in 1.10 >>>>>>> release, and almost all of them happens in Blink planner. >>>>>>> SQL CLI is the most convenient entrypoint for me, I believe many >>>>>>> users will have a better experience If we set Blink planner as default >>>>>>> planner. >>>>>>> >>>>>>> Best, >>>>>>> Leonard >>>>>>> >>>>>>> > 在 2020年1月3日，15:16，Terry Wang 写道： >>>>>>> > >>>>>>> > Since what blink planner can do is a superset of flink planner, >>>>>>> big +1 for changing the default planner to Blink planner from my side. >>>>>>> > >>>>>>> > Best, >>>>>>> > Terry Wang >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> >> 2020年1月3日 15:00，Jark Wu 写道： >>>>>>> >> >>>>>>> >> Hi everyone, >>>>>>> >> >>>>>>> >> In 1.10 release, Flink SQL supports many awesome features and >>>>>>> improvements, >>>>>>> >> including: >>>>>>> >> - support watermark statement and computed column in DDL >>>>>>> >> - fully support all data types in Hive >>>>>>> >> - Batch SQL performance improvements (TPC-DS 7x than Hive MR) >>>>>>> >> - support INSERT OVERWRITE and INSERT PARTITION >>>>>>> >> >>>>>>> >> However, all the features and improvements are only avaiable in >>>>>>> B
[jira] [Created] (FLINK-15469) UpsertStreamTableSink should support new type system
Zhenghua Gao created FLINK-15469:
Summary: UpsertStreamTableSink should support new type system
Key: FLINK-15469
URL: https://issues.apache.org/jira/browse/FLINK-15469
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.11.0
Currently *UpsertStreamTableSink* can only returns TypeInformation of the
requested record, which can't support types with precision and scale, e.g.
TIMESTAMP(p), DECIMAL(p,s).
A proposal is deprecating the *getRecordType* API and adding a
*getRecordDataType* API instead to return the data type of the requested record.
{code:java}
/**
* Returns the requested record type.
*
* @Deprecated This method will be removed in future versions. It's recommended
to use {@link #getRecordDataType()} instead.
*/
@Deprecated
TypeInformation getRecordType();
/*
* Returns the requested record data type.
*/
DataType getRecordDataType();
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15460) planner dependencies won't be necessary for JDBC connector
Zhenghua Gao created FLINK-15460: Summary: planner dependencies won't be necessary for JDBC connector Key: FLINK-15460 URL: https://issues.apache.org/jira/browse/FLINK-15460 Project: Flink Issue Type: Improvement Components: Connectors / HBase, Connectors / JDBC Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15445) JDBC Table Source didn't work for Types with precision (or/and scale)
Zhenghua Gao created FLINK-15445: Summary: JDBC Table Source didn't work for Types with precision (or/and scale) Key: FLINK-15445 URL: https://issues.apache.org/jira/browse/FLINK-15445 Project: Flink Issue Type: Bug Components: Connectors / JDBC Affects Versions: 1.10.0 Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer
Congrats! *Best Regards,* *Zhenghua Gao* On Mon, Dec 16, 2019 at 10:36 AM Biao Liu wrote: > Congrats Zhu Zhu! > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Mon, 16 Dec 2019 at 10:23, Congxian Qiu wrote: > > > Congrats, Zhu Zhu! > > > > Best, > > Congxian > > > > > > aihua li 于2019年12月16日周一 上午10:16写道： > > > > > Congratulations， zhuzhu！ > > > > > > > 在 2019年12月16日，上午10:04，Jingsong Li 写道： > > > > > > > > Congratulations Zhu Zhu! > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Mon, Dec 16, 2019 at 10:01 AM Yang Wang > > > wrote: > > > > > > > >> Congratulations, Zhu Zhu! > > > >> > > > >> wenlong.lwl 于2019年12月16日周一 上午9:56写道： > > > >> > > > >>> Congratulations, Zhu Zhu! > > > >>> > > > >>> On Mon, 16 Dec 2019 at 09:14, Leonard Xu > wrote: > > > >>> > > > >>>> Congratulations, Zhu Zhu ! ! > > > >>>> > > > >>>> Best, > > > >>>> Leonard Xu > > > >>>> > > > >>>>> On Dec 16, 2019, at 07:53, Becket Qin > > wrote: > > > >>>>> > > > >>>>> Congrats, Zhu Zhu! > > > >>>>> > > > >>>>> On Sun, Dec 15, 2019 at 10:26 PM Dian Fu > > > >>> wrote: > > > >>>>> > > > >>>>>> Congrats Zhu Zhu! > > > >>>>>> > > > >>>>>>> 在 2019年12月15日，下午6:23，Zhu Zhu 写道： > > > >>>>>>> > > > >>>>>>> Thanks everyone for the warm welcome! > > > >>>>>>> It's my honor and pleasure to improve Flink with all of you in > > the > > > >>>>>>> community! > > > >>>>>>> > > > >>>>>>> Thanks, > > > >>>>>>> Zhu Zhu > > > >>>>>>> > > > >>>>>>> Benchao Li 于2019年12月15日周日 下午3:54写道： > > > >>>>>>> > > > >>>>>>>> Congratulations!:) > > > >>>>>>>> > > > >>>>>>>> Hequn Cheng 于2019年12月15日周日 上午11:47写道： > > > >>>>>>>> > > > >>>>>>>>> Congrats, Zhu Zhu! > > > >>>>>>>>> > > > >>>>>>>>> Best, Hequn > > > >>>>>>>>> > > > >>>>>>>>> On Sun, Dec 15, 2019 at 6:11 AM Shuyi Chen < > suez1...@gmail.com > > > > > > >>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Congratulations! > > > >>>>>>>>>> > > > >>>>>>>>>> On Sat, Dec 14, 2019 at 7:59 AM Rong Rong < > > walter...@gmail.com> > > > >>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Congrats Zhu Zhu :-) > > > >>>>>>>>>>> > > > >>>>>>>>>>> -- > > > >>>>>>>>>>> Rong > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Sat, Dec 14, 2019 at 4:47 AM tison < > wander4...@gmail.com> > > > >>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Congratulations!:) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best, > > > >>>>>>>>>>>> tison. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> OpenInx 于2019年12月14日周六 下午7:34写道： > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Congrats Zhu Zhu! > > > >>>>>>>>>>>>> > > > >>>>>>>>&
[jira] [Created] (FLINK-15231) Wrong HeapVector in AbstractHeapVector.createHeapColumn
Zhenghua Gao created FLINK-15231: Summary: Wrong HeapVector in AbstractHeapVector.createHeapColumn Key: FLINK-15231 URL: https://issues.apache.org/jira/browse/FLINK-15231 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 For TIMESTAMP WITHOUT TIME ZONE/TIMESTAMP WITH LOCAL TIME ZONE/DECIMAL types, AbstractHeapVector.createHeapColumn generates wrong HeapVectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15213) The conversion between java.sql.Timestamp and long is not asymmetric
Zhenghua Gao created FLINK-15213:
Summary: The conversion between java.sql.Timestamp and long is not
asymmetric
Key: FLINK-15213
URL: https://issues.apache.org/jira/browse/FLINK-15213
Project: Flink
Issue Type: Bug
Reporter: Zhenghua Gao
In Calcite, we use SqlFunctions.toLong(Timestamp) and
SqlFunctions.internalToTimestamp(long) to convert java.sql.Timestmap to
internal long and vice versa. The main logical inside is +/- local time zone
offset.
But in the comments of TimeZone.getOffset(long date), the parameter represents
in milliseconds since January 1, 1970 00:00:00 GMT. It means that there will
one conversion above doesn't satisfy this hypothesis.
This causes many surprise to users:
(1) some Daylight Saving Time changes:
{code:java}
@Test public void testDayLightingSaving() {
TimeZone tz = TimeZone.getDefault();
TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"));
java.sql.Timestamp dst2018Begin = java.sql.Timestamp.valueOf("2018-03-11
03:00:00");
assertThat(dst2018Begin, is(internalToTimestamp(toLong(dst2018Begin;
TimeZone.setDefault(tz);
}{code}
fails with:
{code:java}
java.lang.AssertionError:
Expected: is <2018-03-11 04:00:00.0>
but: was <2018-03-11 03:00:00.0>
Expected :is <2018-03-11 04:00:00.0>
Actual :<2018-03-11 03:00:00.0>{code}
(2) "1900-01-01 00:00:00" Changes in some TimeZone
{code:java}
@Test public void test() {
TimeZone tz = TimeZone.getDefault();
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"));
java.sql.Timestamp ts = java.sql.Timestamp.valueOf("1900-01-01 00:00:00");
assertThat(ts, is(internalToTimestamp(toLong(ts;
TimeZone.setDefault(tz);
}{code}
fails with
{code:java}
java.lang.AssertionError:
Expected: is <1899-12-31 23:54:17.0>
but: was <1900-01-01 00:00:00.0>
Expected :is <1899-12-31 23:54:17.0>
Actual :<1900-01-01 00:00:00.0>
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15151) Use new type system in TableSourceUtil.computeIndexMapping of blink planner
Zhenghua Gao created FLINK-15151: Summary: Use new type system in TableSourceUtil.computeIndexMapping of blink planner Key: FLINK-15151 URL: https://issues.apache.org/jira/browse/FLINK-15151 Project: Flink Issue Type: Bug Components: Table SQL / Planner Reporter: Zhenghua Gao We should use new type system instead of TypeInformation in TableSourceUtil.computeIndexMapping in blink planner -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Drop Kafka 0.8/0.9
+1 for dropping. *Best Regards,* *Zhenghua Gao* On Thu, Dec 5, 2019 at 11:08 AM Dian Fu wrote: > +1 for dropping them. > > Just FYI: there was a similar discussion few months ago [1]. > > [1] > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html#a29997 > > 在 2019年12月5日，上午10:29，vino yang 写道： > > +1 > > jincheng sun 于2019年12月5日周四 上午10:26写道： > >> +1 for drop it, and Thanks for bring up this discussion Chesnay! >> >> Best, >> Jincheng >> >> Jark Wu 于2019年12月5日周四 上午10:19写道： >> >>> +1 for dropping, also cc'ed user mailing list. >>> >>> >>> Best, >>> Jark >>> >>> On Thu, 5 Dec 2019 at 03:39, Konstantin Knauf >>> wrote: >>> >>> > Hi Chesnay, >>> > >>> > +1 for dropping. I have not heard from any user using 0.8 or 0.9 for a >>> long >>> > while. >>> > >>> > Cheers, >>> > >>> > Konstantin >>> > >>> > On Wed, Dec 4, 2019 at 1:57 PM Chesnay Schepler >>> > wrote: >>> > >>> > > Hello, >>> > > >>> > > What's everyone's take on dropping the Kafka 0.8/0.9 connectors from >>> the >>> > > Flink codebase? >>> > > >>> > > We haven't touched either of them for the 1.10 release, and it seems >>> > > quite unlikely that we will do so in the future. >>> > > >>> > > We could finally close a number of test stability tickets that have >>> been >>> > > lingering for quite a while. >>> > > >>> > > >>> > > Regards, >>> > > >>> > > Chesnay >>> > > >>> > > >>> > >>> > -- >>> > >>> > Konstantin Knauf | Solutions Architect >>> > >>> > +49 160 91394525 >>> > >>> > >>> > Follow us @VervericaData Ververica <https://www.ververica.com/> >>> > >>> > >>> > -- >>> > >>> > Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>> > Conference >>> > >>> > Stream Processing | Event Driven | Real Time >>> > >>> > -- >>> > >>> > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>> > >>> > -- >>> > Ververica GmbH >>> > Registered at Amtsgericht Charlottenburg: HRB 158244 B >>> > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >>> > (Tony) Cheng >>> > >>> >> >
Re: [DISCUSS] Disable conversion between TIMESTAMP and Long in parameters and results of UDXs
Since it is unanimously agreed that we should disable conversion between Timestmap and long in parameters and results of UDXs, in PR [1] we will disable it in blink planner. And we will add a release note in FLINK-14599 [2] of this incompatible modification. <https://github.com/apache/flink/pull/10268> [1] https://github.com/apache/flink/pull/10268 [2] https://issues.apache.org/jira/browse/FLINK-14599 *Best Regards,* *Zhenghua Gao* On Sun, Nov 24, 2019 at 8:44 PM Jark Wu wrote: > Hi, > > +1 to disable it in 1.10. I think it's time to disable and correct the > behavior now. > > Also cc'ed user mailing list to have broader audiences. > > Best, > Jark > > On Sat, 23 Nov 2019 at 16:59, Timo Walther wrote: > >> Hi, >> >> +1 for disabling it in the Blink planner. Once FLIP-65 is implemented >> and a UDX is registered with the new >> TableEnvironment.createTemporaryFunction() we will also have the >> possibility to be fully compliant with the new type system because we >> can advertise a new UDF stack with new behavior. >> >> Also the mentioned documentation page will be updated as part of FLIP-65. >> >> Regards, >> Timo >> >> >> On 22.11.19 11:08, Jingsong Li wrote: >> > +1 to disable, It is already introduced by new type system in >> TimestampType. >> > I think it is time to update document too. >> > >> > Best, >> > Jingsong Lee >> > >> > On Fri, Nov 22, 2019 at 6:05 PM Kurt Young wrote: >> > >> >> +1 to disable, we also need to highlight this in 1.10 release notes. >> >> >> >> Best, >> >> Kurt >> >> >> >> >> >> On Fri, Nov 22, 2019 at 5:56 PM Zhenghua Gao wrote: >> >> >> >>> Hi, >> >>> >> >>> I wanted to bring up the discuss of Disable conversion between >> TIMESTAMP >> >>> and Long in parameters and results of UDXs. >> >>> >> >>> Since FLINK-12253[1] introduce the new TimestampType and conversion >> from >> >>> and >> >>> to long is not supported, the UDXs with Long parameters should not >> >> receive >> >>> TIMESTAMP fields and vice versa. >> >>> >> >>> The current situation is we use long as internal representation of >> >>> TIMESTAMP, the legacy planner and blink planner DO NOT DISABLE this >> >>> conversion. Now FLINK-14599[2] would introduce a new internal >> >>> representation of TIMESTAMP and it's time to make a decision to >> DISABLE >> >> it. >> >>> >> >>> In addition, our document[3] recommends UDXs users use long as >> >>> representation of SQL_TIMESTAMP, which is obsolete too. >> >>> >> >>> Please let me know what you think! >> >>> >> >>> [1] https://issues.apache.org/jira/browse/FLINK-12253 >> >>> [2] https://issues.apache.org/jira/browse/FLINK-14599 >> >>> [3] >> >>> >> >>> >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/udfs.html#best-practices-for-implementing-udfs >> >>> >> >>> *Best Regards,* >> >>> *Zhenghua Gao* >> >>> >> >> >> > >> > >> >>
[jira] [Created] (FLINK-14959) Support precision of LocalZonedTimestampType in blink planner
Zhenghua Gao created FLINK-14959: Summary: Support precision of LocalZonedTimestampType in blink planner Key: FLINK-14959 URL: https://issues.apache.org/jira/browse/FLINK-14959 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Remove old WebUI
+1 to drop the old one. *Best Regards,* *Zhenghua Gao* On Thu, Nov 21, 2019 at 8:05 PM Chesnay Schepler wrote: > Hello everyone, > > Flink 1.9 shipped with a new UI, with the old one being kept around as a > backup in case something wasn't working as expected. > > Currently there are no issues indicating any significant problems > (exclusive to the new UI), so I wanted to check what people think about > dropping the old UI for 1.10. > >
[DISCUSS] Disable conversion between TIMESTAMP and Long in parameters and results of UDXs
Hi, I wanted to bring up the discuss of Disable conversion between TIMESTAMP and Long in parameters and results of UDXs. Since FLINK-12253[1] introduce the new TimestampType and conversion from and to long is not supported, the UDXs with Long parameters should not receive TIMESTAMP fields and vice versa. The current situation is we use long as internal representation of TIMESTAMP, the legacy planner and blink planner DO NOT DISABLE this conversion. Now FLINK-14599[2] would introduce a new internal representation of TIMESTAMP and it's time to make a decision to DISABLE it. In addition, our document[3] recommends UDXs users use long as representation of SQL_TIMESTAMP, which is obsolete too. Please let me know what you think! [1] https://issues.apache.org/jira/browse/FLINK-12253 [2] https://issues.apache.org/jira/browse/FLINK-14599 [3] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/udfs.html#best-practices-for-implementing-udfs *Best Regards,* *Zhenghua Gao*
[jira] [Created] (FLINK-14927) Remove LegacyTimestampTypeInfo and LegacyLocalDateTimeTypeInfo when the conversion is not needed
Zhenghua Gao created FLINK-14927: Summary: Remove LegacyTimestampTypeInfo and LegacyLocalDateTimeTypeInfo when the conversion is not needed Key: FLINK-14927 URL: https://issues.apache.org/jira/browse/FLINK-14927 Project: Flink Issue Type: Improvement Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14925) the return type of TO_TIMESTAMP should be Timestamp(9) instead of Timestamp(3)
Zhenghua Gao created FLINK-14925: Summary: the return type of TO_TIMESTAMP should be Timestamp(9) instead of Timestamp(3) Key: FLINK-14925 URL: https://issues.apache.org/jira/browse/FLINK-14925 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14889) TIMESTAMPADD/TIMESTAMPDIFF with microsecond/nanosecond unit lost precision
Zhenghua Gao created FLINK-14889: Summary: TIMESTAMPADD/TIMESTAMPDIFF with microsecond/nanosecond unit lost precision Key: FLINK-14889 URL: https://issues.apache.org/jira/browse/FLINK-14889 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao Since the TimestampAddConvertlet and TimestampDiffConvertlet tread TIMESTAMP as long (with millisecond precision), they lost precision even if the downstream can support microsecond or nanosecond. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14848) BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part
Zhenghua Gao created FLINK-14848: Summary: BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part Key: FLINK-14848 URL: https://issues.apache.org/jira/browse/FLINK-14848 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao For the fixed-length objects, the writer calls setNullAt() to update fixed-length part（which set null bits and initialize fixed-length part with 0; For the variable-length objects, the writer calls setNullAt to update fixed-length part and need to assign & initialize variable-length part -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14810) It's weird that copy julianDateFloor from DateTimeUtils and change the implementation
Zhenghua Gao created FLINK-14810: Summary: It's weird that copy julianDateFloor from DateTimeUtils and change the implementation Key: FLINK-14810 URL: https://issues.apache.org/jira/browse/FLINK-14810 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Zhenghua Gao In SqlDateTimeUtils we copied *julianToLocalDate* from DateTimeUtils and changed the implementation. It's weird to use an *entirely* *different* implementation to do the same thing. One possible improvement of the new one is support TimeUnitRange.QUARTER. But the logic to calculate year/month/day is totally different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14806) Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter
Zhenghua Gao created FLINK-14806: Summary: Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter Key: FLINK-14806 URL: https://issues.apache.org/jira/browse/FLINK-14806 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 Since FLINK-14080 introduce a new representation of TimestampType, the binary format should add timestamp related interface to set/get TimestampType objects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14805) Remove unixDateCeil related code
Zhenghua Gao created FLINK-14805: Summary: Remove unixDateCeil related code Key: FLINK-14805 URL: https://issues.apache.org/jira/browse/FLINK-14805 Project: Flink Issue Type: Improvement Components: Table SQL / Legacy Planner, Table SQL / Planner Reporter: Zhenghua Gao FLINK-11935 removed DateTimeUtils and add unixDateCeil related code to fix CEIL(date) (CALCITE-3199). CALCITE-3199 has merged to avatica-1.16.0 and we can remove the copied code when we upgraded to avatica-1.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14764) The final Explicit/Implicit conversion matrix we should support in our planner
Zhenghua Gao created FLINK-14764: Summary: The final Explicit/Implicit conversion matrix we should support in our planner Key: FLINK-14764 URL: https://issues.apache.org/jira/browse/FLINK-14764 Project: Flink Issue Type: Improvement Components: Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 The SQL standard defines the cast specification with an explicit conversion matrix (SQL 2011 Part 2 Section 6.13 Syntax Rules 6)). But neither Legacy planner nor blink planner would follow that. IMO we should determine a final Explicit/Implicit conversion matrix before 1.10 (at least in blink planner). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14737) HiveTypeUtil.toFlinkPrimitiveType returns wrong Timestamp(6) instead of Timestamp(9)
Zhenghua Gao created FLINK-14737: Summary: HiveTypeUtil.toFlinkPrimitiveType returns wrong Timestamp(6) instead of Timestamp(9) Key: FLINK-14737 URL: https://issues.apache.org/jira/browse/FLINK-14737 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Zhenghua Gao Hive's Timestamp type holds nanosecond's precision, and when transfer to Flink typesystem, it's should be Timestamp(9). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14696) Support precision of TimestampType in built-in SQL functions and operators
Zhenghua Gao created FLINK-14696: Summary: Support precision of TimestampType in built-in SQL functions and operators Key: FLINK-14696 URL: https://issues.apache.org/jira/browse/FLINK-14696 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Zhenghua Gao Many built-in SQL functions and operators use long as internal representation of Timestamp type and only support millisecond precision. This ticket will check fix it and let them support nanosecond precision. The related SQL functions and operators are: (To Be Confirmed) -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC
Congratulations Jark! *Best Regards,* *Zhenghua Gao* On Mon, Nov 11, 2019 at 8:54 AM SHI Xiaogang wrote: > Congratulations, Jark. > You make a great job of improving the community. > > Regards, > Xiaogang > > Benchao Li 于2019年11月10日周日 下午4:14写道： > > > Congratulations, Jark! > > > > Yun Tang 于2019年11月10日 周日下午3:25写道： > > > > > Congratulations, Jark > > > > > > Best > > > Yun Tang > > > > > > On 11/10/19, 10:40 AM, "vino yang" wrote: > > > > > > Congratulations, Jark. > > > > > > Best, > > > Vino > > > > > > Leonard Xu 于2019年11月8日周五 下午8:40写道： > > > > > > > Congratulations, Jark. > > > > Thanks for your contribution and help. > > > > > > > > Best, > > > > Leonard Xu > > > > > > > > > On 2019年11月8日, at 下午6:37, Yun Gao > > > > wrote: > > > > > > > > > > Congratulations Jark！ > > > > > > > > > > Best, > > > > > Yun > > > > > > > > > > > > > > > > > -- > > > > > From:wenlong.lwl > > > > > Send Time:2019 Nov. 8 (Fri.) 18:31 > > > > > To:dev > > > > > Subject:Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC > > > > > > > > > > Congratulations Jark, well deserved! > > > > > > > > > > > > > > > Best, > > > > > Wenlong Lyu > > > > > > > > > > On Fri, 8 Nov 2019 at 18:22, tison > wrote: > > > > > > > > > >> Congrats Jark! > > > > >> > > > > >> Best, > > > > >> tison. > > > > >> > > > > >> > > > > >> Jingsong Li 于2019年11月8日周五 下午6:08写道： > > > > >> > > > > >>> Congratulations to Jark. > > > > >>> Jark has really contributed a lot to the table layer with a > > long > > > time. > > > > >> Well > > > > >>> deserved. > > > > >>> > > > > >>> Best, > > > > >>> Jingsong Lee > > > > >>> > > > > >>> On Fri, Nov 8, 2019 at 6:05 PM Yu Li > wrote: > > > > >>> > > > > >>>> Congratulations Jark! Well deserved! > > > > >>>> > > > > >>>> Best Regards, > > > > >>>> Yu > > > > >>>> > > > > >>>> > > > > >>>> On Fri, 8 Nov 2019 at 17:55, OpenInx > > wrote: > > > > >>>> > > > > >>>>> Congrats Jark ! Well deserve. > > > > >>>>> > > > > >>>>> On Fri, Nov 8, 2019 at 5:53 PM Paul Lam < > > paullin3...@gmail.com > > > > > > > > >> wrote: > > > > >>>>> > > > > >>>>>> Congrats Jark! > > > > >>>>>> > > > > >>>>>> Best, > > > > >>>>>> Paul Lam > > > > >>>>>> > > > > >>>>>>> 在 2019年11月8日，17:51，jincheng sun < > sunjincheng...@gmail.com> > > > 写道： > > > > >>>>>>> > > > > >>>>>>> Hi all, > > > > >>>>>>> > > > > >>>>>>> On behalf of the Flink PMC, I'm happy to announce that > Jark > > > Wu is > > > > >>> now > > > > >>>>>>> part of the Apache Flink Project Management Committee > > (PMC). > > > > >>>>>>> > > > > >>>>>>> Jark has been a committer since February 2017. He has > been > > > very > > > > >>>> active > > > > >>>>> on > > > > >>>>>>> Flink's Table API / SQL component, as well as frequently > > > helping > > > > >>>>>>> manage/verify/vote releases. He has been writing many > blogs > > > about > > > > >>>>> Flink, > > > > >>>>>>> also driving the translation work of Flink website and > > > > >>> documentation. > > > > >>>>> He > > > > >>>>>> is > > > > >>>>>>> very active in China community as he gives talks about > > Flink > > > at > > > > >>> many > > > > >>>>>> events > > > > >>>>>>> in China. > > > > >>>>>>> > > > > >>>>>>> Congratulations & Welcome Jark! > > > > >>>>>>> > > > > >>>>>>> Best, > > > > >>>>>>> Jincheng (on behalf of the Flink PMC) > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> Best, Jingsong Lee > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: libenc...@gmail.com; libenc...@pku.edu.cn > > >
[jira] [Created] (FLINK-14650) Thread safety issue in the piece of code example of dev/stream/testing document
Zhenghua Gao created FLINK-14650: Summary: Thread safety issue in the piece of code example of dev/stream/testing document Key: FLINK-14650 URL: https://issues.apache.org/jira/browse/FLINK-14650 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Zhenghua Gao As mentioned by Gilles in user ML[1], the piece of code example has thread safety issue. One possibility is use Collections.synchronizedList() to create a thread-safety list and remove the synchronized keyword. [1] [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Documentation-issue-maybe-td30929.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14599) Support precision of TimestampType
Zhenghua Gao created FLINK-14599: Summary: Support precision of TimestampType Key: FLINK-14599 URL: https://issues.apache.org/jira/browse/FLINK-14599 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Zhenghua Gao Since FLINK-14080 introduced an internal representation(SqlTimestamp) of TimestampType with precision. This subtask will replace current long with SqlTimestamp, and let blink planner support precision of TimestampType. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Move flink-orc to flink-formats from flink-connectors
+1 (non-binding) *Best Regards,* *Zhenghua Gao* On Wed, Oct 30, 2019 at 12:05 PM Jingsong Li wrote: > Hi all: > > We already have the parent model of formats. we have put other > formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, > flink-sequence-file) to flink-formats. flink-orc is a format too. So we can > move it to flink-formats. > > In theory, there should be no compatibility problem, only the parent model > needs to be changed, and no other changes are needed. > > I would like to start the vote for it. The vote will be open for at least > 72 hours. > > Discuss thread: [1] > > [1] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-flink-orc-to-flink-formats-from-flink-connectors-td34438.html > > -- > Best, Jingsong Lee >
Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors
+1 to move flink-orc to flink-formats. Since we have put other file-based formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, flink-sequence-file) in flink-formats, flink-orc should be the same. *Best Regards,* *Zhenghua Gao* On Tue, Oct 29, 2019 at 11:10 AM Jingsong Lee wrote: > Hi devs, > > I found that Orc is still in connectors, but we already have the parent > model of formats. > - Is it better to put flink-orc in flink-formats? > - Is there a compatibility issue with the move? Looks like only the parent > model needs to be changed, and no other changes are needed. > > -- > Best, Jingsong Lee >
Re: How to specify a test to run in Flink?
Actually it's not a Flink problem. For single module project, you can run "mvn -Dtest=YOUR_TEST test" to run a single test. For multiple modules project, you can use "-pl sub-module" to specify the module which your test belongs to( mvn -Dtest=YOUR_TEST -pl YOUR_MODULE test), OR just CD to your module directory and run "mvn -Dtest=YOUR_TEST test" *Best Regards,* *Zhenghua Gao* On Tue, Oct 29, 2019 at 10:19 AM 朱国梁 wrote: > > Hi community! I have a problem that I cannot solve by google. > > > I am trying to specify a test to run using maven. > > > mvn clean test -Dtest=DistributedCacheTest > > > The result says that: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) > on project force-shading: No tests were executed! (Set > -DfailIfNoTests=false to ignore this error.) -> [Help 1] > > > > > > -- > > --- > Best > zgl > --- > > > >
Re: [VOTE] FLIP-70: Flink SQL Computed Column Design
+1 (non-binding) *Best Regards,* *Zhenghua Gao* On Mon, Oct 28, 2019 at 2:26 PM Danny Chan wrote: > Hi all, > > I would like to start the vote for FLIP-70[1] which is discussed and > reached consensus in the discussion thread[2]. > > The vote will be open for at least 72 hours. I'll try to close it by > 2019-10-31 18:00 UTC, unless there is an objection or not enough votes. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design > [2] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-70-Support-Computed-Column-for-Flink-SQL-td33126.html > > Best, > Danny Chan >
Re: [ANNOUNCE] Becket Qin joins the Flink PMC
Congratulations, Becket! *Best Regards,* *Zhenghua Gao* On Tue, Oct 29, 2019 at 10:34 AM Yun Gao wrote: > Congratulations Becket! > > Best, > Yun > > > -- > From:Jingsong Li > Send Time:2019 Oct. 29 (Tue.) 10:23 > To:dev > Subject:Re: [ANNOUNCE] Becket Qin joins the Flink PMC > > Congratulations Becket! > > Best, > Jingsong Lee > > On Tue, Oct 29, 2019 at 10:18 AM Terry Wang wrote: > > > Congratulations, Becket! > > > > Best, > > Terry Wang > > > > > > > > > 2019年10月29日 10:12，OpenInx 写道： > > > > > > Congratulations Becket! > > > > > > On Tue, Oct 29, 2019 at 10:06 AM Zili Chen > wrote: > > > > > >> Congratulations Becket! > > >> > > >> Best, > > >> tison. > > >> > > >> > > >> Congxian Qiu 于2019年10月29日周二 上午9:53写道： > > >> > > >>> Congratulations Becket! > > >>> > > >>> Best, > > >>> Congxian > > >>> > > >>> > > >>> Wei Zhong 于2019年10月29日周二 上午9:42写道： > > >>> > > >>>> Congratulations Becket! > > >>>> > > >>>> Best, > > >>>> Wei > > >>>> > > >>>>> 在 2019年10月29日，09:36，Paul Lam 写道： > > >>>>> > > >>>>> Congrats Becket! > > >>>>> > > >>>>> Best, > > >>>>> Paul Lam > > >>>>> > > >>>>>> 在 2019年10月29日，02:18，Xingcan Cui 写道： > > >>>>>> > > >>>>>> Congratulations, Becket! > > >>>>>> > > >>>>>> Best, > > >>>>>> Xingcan > > >>>>>> > > >>>>>>> On Oct 28, 2019, at 1:23 PM, Xuefu Z wrote: > > >>>>>>> > > >>>>>>> Congratulations, Becket! > > >>>>>>> > > >>>>>>> On Mon, Oct 28, 2019 at 10:08 AM Zhu Zhu > > >> wrote: > > >>>>>>> > > >>>>>>>> Congratulations Becket! > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> Zhu Zhu > > >>>>>>>> > > >>>>>>>> Peter Huang 于2019年10月29日周二 > 上午1:01写道： > > >>>>>>>> > > >>>>>>>>> Congratulations Becket Qin! > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Best Regards > > >>>>>>>>> Peter Huang > > >>>>>>>>> > > >>>>>>>>> On Mon, Oct 28, 2019 at 9:19 AM Rong Rong > > > >>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Congratulations Becket!! > > >>>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>> Rong > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Oct 28, 2019, 7:53 AM Jark Wu > wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Congratulations Becket! > > >>>>>>>>>>> > > >>>>>>>>>>> Best, > > >>>>>>>>>>> Jark > > >>>>>>>>>>> > > >>>>>>>>>>> On Mon, 28 Oct 2019 at 20:26, Benchao Li < > libenc...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Congratulations Becket. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Dian Fu 于2019年10月28日周一 下午7:22写道： > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Congrats, Becket. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> 在 2019年10月28日，下午6:07，Fabian Hueske > 写道： > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I'm happy to announce that Becket Qin has joined the Flink > > >>> PMC. > > >>>>>>>>>>>>>> Let's congratulate and welcome Becket as a new member of > the > > >>>>>>>>> Flink > > >>>>>>>>>>> PMC! > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>> Fabian > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> -- > > >>>>>>>>>>>> > > >>>>>>>>>>>> Benchao Li > > >>>>>>>>>>>> School of Electronics Engineering and Computer Science, > Peking > > >>>>>>>>>> University > > >>>>>>>>>>>> Tel:+86-15650713730 > > >>>>>>>>>>>> Email: libenc...@gmail.com; libenc...@pku.edu.cn > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Xuefu Zhang > > >>>>>>> > > >>>>>>> "In Honey We Trust!" > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>> > > >> > > > > > > -- > Best, Jingsong Lee > >
[jira] [Created] (FLINK-14535) Fix distinct key type for DecimalType in DistinctInfo
Zhenghua Gao created FLINK-14535: Summary: Fix distinct key type for DecimalType in DistinctInfo Key: FLINK-14535 URL: https://issues.apache.org/jira/browse/FLINK-14535 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.1 Reporter: Zhenghua Gao Fix For: 1.10.0 DecimalType in DistinctInfo bridged to wrong external BigDecimal type, which causes failures count distinct on decimal type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Confluence permission for FLIP creation
Many Thanks for your help! *Best Regards,* *Zhenghua Gao* On Wed, Sep 25, 2019 at 9:33 AM jincheng sun wrote: > I have given you edit permission, could you please re-login and check it. > :) > > Zhenghua Gao 于2019年9月24日周二 下午8:13写道： > > > Hi all, > > > > I'm proposing to support view in Flink SQL and I have a initial design > doc > > which i want to convert to a FLIP. It would be great if anyone who can > > grant me the write permission to Confluence. My Confluence ID is: docete > > > > I will really appreciate if any of you can help me with this. > > > > *Best Regards,* > > *Zhenghua Gao* > > >
Confluence permission for FLIP creation
Hi all, I'm proposing to support view in Flink SQL and I have a initial design doc which i want to convert to a FLIP. It would be great if anyone who can grant me the write permission to Confluence. My Confluence ID is: docete I will really appreciate if any of you can help me with this. *Best Regards,* *Zhenghua Gao*
Re: Can I do a lookup in FlinkSQL?
The lookup fashion Temporal Join[1] should be a solution for your case and there is an ITCase as an example[2] [1] https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/sources/LookupableTableSource.java [2] https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/AsyncLookupJoinITCase.scala *Best Regards,* *Zhenghua Gao* On Mon, Sep 16, 2019 at 9:23 PM srikanth flink wrote: > Hi there, > > I'm working with streaming in FlinkSQL. I've two tables created one with > dynamic stream and the other a periodic updates. > I would like to keep the periodic table a static(but updates with new data > every day or so by flushing the old), So at any point of time the static > table should contain new set of data. > With dynamic table being populated with stream data, could I do a lookup on > a column of static table to find if the value exists. > > This is what I have done: > dynamic table: sourceKafka > static table: badips > > Trying to build a list, kind of using ROW() function and done. From dynamic > table, trying to lookup into the list if the value exists. > Query: INSERT INTO sourceKafkaMalicious select s.* from sourceKafka as s > where s.`source.ip` OR s.`destination.ip` IN (select ROW(ip) from badips); > Resonse: > [INFO] Submitting SQL update statement to the cluster... > [ERROR] Could not execute SQL statement. Reason: > org.apache.calcite.sql.validate.SqlValidatorException: Values passed to IN > operator must have compatible types > > Is it possible to solve my use case? If so, where am I going wrong? > > Thanks > Srikanth >
[DISCUSS] FLIP-71 - E2E View support in Flink SQL
Hi folks, In umbrella task FLINK-10232 we have introduced CREATE/DROP VIEW grammar in our module flink-sql-parser. But we don't support view objects in neither blink planner nor old planner. I'd like to kick off a discussion on end to end view support in Flink SQL in blink planner. It's helpful to improve the usability of the framework for SQL users. https://docs.google.com/document/d/14bx0t8wYH7_o4ChNkDoBFGn-i0T-Q7kUiOFvDd13_Fk/edit#heading=h.m031smarjj9p In short, it: - support define views and store them in catalog - support drop view definitions from catalog - support query views - support other view related DDLs Any comments and feedbacks are welcome and appreciated. Thanks. *Best Regards,* *Zhenghua Gao*
Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module
Thanks david for pushing this forward. I have one concern about temporary objects and non-persistent catalog(e.g., GenericInMemoryCatalog). In SQL, temporary objects exist at the session level. They are only visible to the session in which they were created and are automatically dropped when that session logs off. So in that sense, all objects in non-persistent catalogs should be temporary. My concern is: How does the non-persistent catalog work with temporary objects? *Best Regards,* *Zhenghua Gao* On Wed, Sep 4, 2019 at 10:20 PM Dawid Wysakowicz wrote: > Hi all, > > As part of FLIP-30 > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs> > a Catalog API was introduced that enables storing table meta objects > permanently. At the same time the majority of current APIs create temporary > objects that cannot be serialized. We should clarify the creation of meta > objects (tables, views, functions) in a unified way. > > Another current problem in the API is that all the temporary objects are > stored in a special built-in catalog, which is not very intuitive for many > users, as they must be aware of that catalog to reference temporary objects. > > Lastly, different APIs have different ways of providing object paths: > >- String path…, >- String path, String pathContinued… >- String name > > We should choose one approach and unify it across all APIs. > > I suggest a FLIP to address the above issues. > > Looking forward to your opinions. > > FLIP link: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module >
Re: [DISCUSS] Best practice to run flink on kubernetes
Thanks Yang for bringing this up. I think option1 is very useful for early
adopters.
People do not know much about k8s and can easily set up on minikube to have
a taste.
For option2 and option3, i prefer option3 because i am familiar yarn and
don't have much concept of k8s.
And there is some doube about starting a session cluster in option3:
> ./bin/kubernetes-session.sh -d -n 2 -tm 512 -s 4 -nm flink-session-example
> -i flink:latest -kD kubernetes.service.exposed.type=NODE_PORT
Is the -n option means number of TaskManager?
Do we pre-running taskmanager pods or requesting and launching taskmanager
pods dynamically?
*Best Regards,*
*Zhenghua Gao*
On Fri, Aug 9, 2019 at 9:12 PM Yang Wang wrote:
> Hi all,
>
> Currently cloud native architectures has been introduced to many companies
> in production. They use kubernetes to run deep learning, web server, etc.
> If we could deploy the per-job/session flink cluster on kubernetes to make
> it mix-run with other workloads, the cluster resource utilization will be
> better. Also many kubernetes users are more easier to have a taste on the
> flink.
>
> By now we have three options to run flink jobs on k8s.
>
> [1]. Create jm/tm/service yaml and apply, then you will get a flink
> standalone cluster on k8s. Use flink run to submit job to the existed flink
> cluster. Some companies may have their own deploy system to manage the
> flink cluster.
>
> [2]. Use flink-k8s-operator to manage multiple flink clusters, including
> session and perjob. It could manage the complete deployment lifecycle of
> the application. I think this option is really easy to use for the k8s
> users. They are familiar with k8s-opertor, kubectl and other tools of k8s.
> They could debug and run the flink cluster just like other k8s
> applications.
>
> [3]. Natively integration with k8s, use the flink run or
> kubernetes-session.sh to start a flink cluster. It is very similar to
> submitting an flink cluster to Yarn. KubernetesClusterDescriptor talks to
> k8s api server to start a flink master deployment of 1.
> KubernetesResourceManager dynamically allocates resource from k8s to start
> task manager as demand. This option is very easy for flink users to get
> started. In the simplest case, we just need to update the '-m yarn-cluster'
> to -m '-m kubernetes-cluster'.
>
> We have make an internal implementation of option [3] and use it in
> production. After fully tested, we hope to contribute it to the community.
> Now we want to get some feedbacks about the three options. Any comments are
> welcome.
>
>
> > What do we need to prepare when start a flink cluster on k8s using native
> integration?
>
> Download the flink release binary and create the ~/.kube/config file
> corresponding to the k8s cluster. It is all what you need.
>
>
> > Flink Session cluster
>
> * start a session cluster
>
> ./bin/kubernetes-session.sh -d -n 2 -tm 512 -s 4 -nm flink-session-example
> -i flink:latest -kD kubernetes.service.exposed.type=NODE_PORT
>
> * You will get an address to submit job, specify it through ’-ksa’ option
>
> ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-session-example
> -ksa {x.x.x.x:12345} examples/streaming/WindowJoin.jar
>
>
> > Flink Job Cluster
>
> * running with official flink image
>
> ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1
> -ki flink:latest examples/streaming/WindowJoin.jar
>
> * running with user image
>
> ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1
> -ki flink-user:latest examples/streaming/WindowJoin.jar
>
>
>
> [1].
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html
>
> [2].https://github.com/lyft/flinkk8soperator
>
> [3].
>
> https://docs.google.com/document/d/1Zmhui_29VASPcBOEqyMWnF3L6WEWZ4kedrCqya0WaAk/edit#
>
[jira] [Created] (FLINK-13665) decimal(p, s) where p is less than s should be illegal
Zhenghua Gao created FLINK-13665:
Summary: decimal(p, s) where p is less than s should be illegal
Key: FLINK-13665
URL: https://issues.apache.org/jira/browse/FLINK-13665
Project: Flink
Issue Type: Bug
Components: Table SQL / Legacy Planner
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.10.0
e.g.: {{cast(42.345 as decimal(2, 3)) should get a ValidationException}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (FLINK-13664) "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING
Zhenghua Gao created FLINK-13664:
Summary: "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER
instend of SqlTypeFamily.STRING
Key: FLINK-13664
URL: https://issues.apache.org/jira/browse/FLINK-13664
Project: Flink
Issue Type: Bug
Reporter: Zhenghua Gao
Both planners do not support MD5(binary), this should fast fail at validate
phrase and give users more meaningful messages.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (FLINK-13651) table api not support cast to decimal with precision and scale
Zhenghua Gao created FLINK-13651: Summary: table api not support cast to decimal with precision and scale Key: FLINK-13651 URL: https://issues.apache.org/jira/browse/FLINK-13651 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao could reproduce in ScalarFunctionsTest: `testAllApis( 'f31.cast(DataTypes.DECIMAL(38, 18)).truncate(2), "f31.cast(DECIMAL(10, 10)).truncate(2)", "truncate(cast(f31 as decimal(38, 18)), 2)", "-0.12")` A possible reason is LookupCallResolver treat decimal(38, 18) as a function call. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13547) Verify and correct string function's semantic for Blink planner
Zhenghua Gao created FLINK-13547: Summary: Verify and correct string function's semantic for Blink planner Key: FLINK-13547 URL: https://issues.apache.org/jira/browse/FLINK-13547 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13523) Verify and correct arithmetic function's semantic for Blink planner
Zhenghua Gao created FLINK-13523: Summary: Verify and correct arithmetic function's semantic for Blink planner Key: FLINK-13523 URL: https://issues.apache.org/jira/browse/FLINK-13523 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13522) Verify and correct builtin function's semantic for Blink planner
Zhenghua Gao created FLINK-13522: Summary: Verify and correct builtin function's semantic for Blink planner Key: FLINK-13522 URL: https://issues.apache.org/jira/browse/FLINK-13522 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13494) Blink planner changes source parallelism which causes stream SQL e2e test fails
Zhenghua Gao created FLINK-13494: Summary: Blink planner changes source parallelism which causes stream SQL e2e test fails Key: FLINK-13494 URL: https://issues.apache.org/jira/browse/FLINK-13494 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13394) secure MapR repo URL is not work in E2E crontab builds
Zhenghua Gao created FLINK-13394: Summary: secure MapR repo URL is not work in E2E crontab builds Key: FLINK-13394 URL: https://issues.apache.org/jira/browse/FLINK-13394 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 [FLINK-12578|https://issues.apache.org/jira/browse/FLINK-12578] [FLINK-12578|http://example.com/] intros https URL for MapR, but this causes fails on Travis for some reason. travis_watchdog.sh and travis_controller.sh are fixed by unsafe-mapr-repo profile, but nightly.sh is not fixed. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13377) Streaming SQL e2e test failed on travis
Zhenghua Gao created FLINK-13377: Summary: Streaming SQL e2e test failed on travis Key: FLINK-13377 URL: https://issues.apache.org/jira/browse/FLINK-13377 Project: Flink Issue Type: Bug Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 This is an instance: [https://api.travis-ci.org/v3/job/562011491/log.txt] == Running 'Streaming SQL end-to-end test' == TEST_DATA_DIR: /home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314 Flink dist directory: /home/travis/build/apache/flink/flink-dist/target/flink-1.9-SNAPSHOT-bin/flink-1.9-SNAPSHOT Starting cluster. Starting standalonesession daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Dispatcher REST endpoint is up. [INFO] 1 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 2 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 3 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting execution of program Program execution finished Job with JobID 7c7b66dd4e8dc17e229700b1c746aba6 has finished. Job Runtime: 77371 ms cat: '/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/.part-*': No such file or directory cat: '/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/part-*': No such file or directory FAIL StreamSQL: Output hash mismatch. Got d41d8cd98f00b204e9800998ecf8427e, expected b29f14ed221a936211202ff65b51ee26. head hexdump of actual: Stopping taskexecutor daemon (pid: 9983) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping standalonesession daemon (pid: 8088) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 21571), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 22154), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 22595), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 30622), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 3850), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 4405), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 4839), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 8531) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 9077) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 9518) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [FAIL] Test script contains errors. Checking of logs skipped. [FAIL] 'Streaming SQL end-to-end test' failed after 1 minutes and 51 seconds! Test exited with exit code 1 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13374) flink-table-planner-blink compile failed for scala-2.12 on travis
Zhenghua Gao created FLINK-13374: Summary: flink-table-planner-blink compile failed for scala-2.12 on travis Key: FLINK-13374 URL: https://issues.apache.org/jira/browse/FLINK-13374 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 Here is a instance: [https://api.travis-ci.org/v3/job/562043336/log.txt] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13302) DateTimeUtils.unixDateCeil returns the same value as unixDateFloor does
Zhenghua Gao created FLINK-13302: Summary: DateTimeUtils.unixDateCeil returns the same value as unixDateFloor does Key: FLINK-13302 URL: https://issues.apache.org/jira/browse/FLINK-13302 Project: Flink Issue Type: Bug Components: Table SQL / Legacy Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Assignee: Zhenghua Gao Fix For: 1.9.0, 1.10.0 Internally, unixDateCeil & unixDateFloor call julianDateFloor and pass a boolean parameter to differentiate them. But unixDateCeil passes wrong parameter value and returns the same value as unixDateFloor does. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13280) Revert blink changes in DateTimeUtils, and keep it same as flink version.
Zhenghua Gao created FLINK-13280: Summary: Revert blink changes in DateTimeUtils, and keep it same as flink version. Key: FLINK-13280 URL: https://issues.apache.org/jira/browse/FLINK-13280 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 This class have some diff between flink/blink planner: * Blink intros some constants (e.g., MICROS_PER_DAY, SECONDS_PER_DAY), inner use, it does not matter. * Blink intros a function unixDateTimeToString (new) * Blink changes the behavior of some function * dateStringToUnixDate: only used in test & codegen now, can be moved into another util class * timeStringToUnixDate: only used in test & codegen now, can be moved into another util class * Blink intros USER TimeZone, but now it’s always UTC TimeZone, so it does not matter -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13105) Add documentation for blink planner's built-in functions
Zhenghua Gao created FLINK-13105: Summary: Add documentation for blink planner's built-in functions Key: FLINK-13105 URL: https://issues.apache.org/jira/browse/FLINK-13105 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Zhenghua Gao Assignee: Zhenghua Gao Fix For: 1.9.0 Blink planner intros some built-in functions which need to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12999) Can't generate valid execution plan for "SELECT uuid() FROM VALUES(1) T(a)"
Zhenghua Gao created FLINK-12999:
Summary: Can't generate valid execution plan for "SELECT uuid()
FROM VALUES(1) T(a)"
Key: FLINK-12999
URL: https://issues.apache.org/jira/browse/FLINK-12999
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Zhenghua Gao
The ERROR message is:
org.apache.flink.table.api.TableException: Cannot generate a valid execution
plan for the given query:
LogicalSink(fields=[EXPR$0])
+- LogicalProject(EXPR$0=[UUID()])
+- LogicalValues(tuples=[[\{ 1, 2, 3 }]])
This exception indicates that the query uses an unsupported SQL feature.
Please check the documentation for the set of currently supported SQL features.
at
org.apache.flink.table.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:72)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.optimizeTree(BatchCommonSubGraphBasedOptimizer.scala:82)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.org$apache$flink$table$plan$optimize$BatchCommonSubGraphBasedOptimizer$$optimizeBlock(BatchCommonSubGraphBasedOptimizer.scala:51)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:39)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:39)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.doOptimize(BatchCommonSubGraphBasedOptimizer.scala:39)
at
org.apache.flink.table.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:65)
at
org.apache.flink.table.api.TableEnvironment.optimize(TableEnvironment.scala:251)
at
org.apache.flink.table.api.TableEnvironment.compileToExecNodePlan(TableEnvironment.scala:200)
at
org.apache.flink.table.api.TableEnvironment.compile(TableEnvironment.scala:184)
at
org.apache.flink.table.api.TableEnvironment.generateStreamGraph(TableEnvironment.scala:155)
at
org.apache.flink.table.api.BatchTableEnvironment.execute(BatchTableEnvironment.scala:93)
at
org.apache.flink.table.api.TableEnvironment.execute(TableEnvironment.scala:136)
at
org.apache.flink.table.runtime.utils.BatchTableEnvUtil$.collect(BatchTableEnvUtil.scala:55)
at
org.apache.flink.table.runtime.utils.TableUtil$.collectSink(TableUtil.scala:60)
at org.apache.flink.table.runtime.utils.TableUtil$.collect(TableUtil.scala:41)
at
org.apache.flink.table.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:308)
at
org.apache.flink.table.runtime.utils.BatchTestBase.check(BatchTestBase.scala:164)
at
org.apache.flink.table.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:103)
at
org.apache.flink.table.runtime.batch.sql.ValuesITCase.test(ValuesITCase.scala:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4Class
[jira] [Created] (FLINK-12845) Execute multiple statements in command line or sql script file
Zhenghua Gao created FLINK-12845: Summary: Execute multiple statements in command line or sql script file Key: FLINK-12845 URL: https://issues.apache.org/jira/browse/FLINK-12845 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao User may copy multiple statements and paste them on command line GUI of SQL Client, or User may pass a script file(using SOURCE command or -f option), we should parse and execute them one by one(like other sql cli applications) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12828) Support -f option with a sql script file as input
Zhenghua Gao created FLINK-12828: Summary: Support -f option with a sql script file as input Key: FLINK-12828 URL: https://issues.apache.org/jira/browse/FLINK-12828 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao We expect user to run a script file directly on the command line. Something like: sql-client embedded -f myscript.sql, which will execute the given file without entering interactive mode -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12819) Reuse TableEnvironment between different SQL statements
Zhenghua Gao created FLINK-12819: Summary: Reuse TableEnvironment between different SQL statements Key: FLINK-12819 URL: https://issues.apache.org/jira/browse/FLINK-12819 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao We have introduced catalogs to store catalog object(tables, views etc). And the catalogs are tied to TableEnvironment, So we need to reuse TableEnvironment so the previously registered tables and views are available(suppose we use an InMemory catalog). BTW, reuse TableEnvironment is more resource and time saving. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12814) Support a traditional and scrolling view of result (non-interactive)
Zhenghua Gao created FLINK-12814: Summary: Support a traditional and scrolling view of result (non-interactive) Key: FLINK-12814 URL: https://issues.apache.org/jira/browse/FLINK-12814 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Affects Versions: 1.8.0 Reporter: Zhenghua Gao Assignee: Zhenghua Gao Attachments: image-2019-06-12-16-11-06-070.png In table mode, we want to introduce a non-interactive view (so-called FinalizedResult), which submit SQL statements(DQLs) in attach mode with a user defined timeout, fetch results until the job finished/failed/timeout or interrupted by user(Ctrl+C), and output them in a non-interactive way (the behavior in change-log mode is under discussion) !image-2019-06-12-16-11-06-070.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8045) Add Internal DATE/TIME/TIMESTAMP as internal representation of DATE/TIME/TIMESTAMP
Zhenghua Gao created FLINK-8045: --- Summary: Add Internal DATE/TIME/TIMESTAMP as internal representation of DATE/TIME/TIMESTAMP Key: FLINK-8045 URL: https://issues.apache.org/jira/browse/FLINK-8045 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Zhenghua Gao Assignee: Zhenghua Gao Currently DATE/TIME/TIMESTAMP have internal representation. Such as Date is represented as Int internal. This feature may improve performance processing DATE/TIME/TIMESTAMP data. But I found there is a LIMITATION: internal representation exists only within one operator. We transfer DATE/TIME/TIMESTAMP objects between operators. I think we could treat DATE/TIME/TIMESTAMP as internal representation in the whole job, and cast them to java.sql.* as needed(UDF/UDTF/OUTPUT) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6173) flink-table not pack-in com.fasterxml.jackson.* in after #FLINK-5414
Zhenghua Gao created FLINK-6173:
---
Summary: flink-table not pack-in com.fasterxml.jackson.* in after
#FLINK-5414
Key: FLINK-6173
URL: https://issues.apache.org/jira/browse/FLINK-6173
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Reporter: Zhenghua Gao
Currently, flink-table will pack-in com.fasterxml.jackson.* and rename them to
org.apache.flink.shaded.calcite.com.fasterxml.jackson.*
If a project depends on flink-table, and uses fasterxml as follows(function
explain uses fasterxml indirectly):
```
object WordCountWithTable {
def main(args: Array[String]): Unit = {
// set up execution environment
val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = TableEnvironment.getTableEnvironment(env)
val input = env.fromElements(WC("hello", 1), WC("hello", 1), WC("ciao", 1))
val expr = input.toTable(tEnv)
val result = expr
.groupBy('word)
.select('word, 'frequency.sum as 'frequency)
.filter('frequency === 2)
println(tEnv.explain(result))
result.toDataSet[WC].print()
}
case class WC(word: String, frequency: Long)
}
```
It actually uses org.apache.flink.shaded.calcite.com.fasterxml.jackson.*
I found after FLINK-5414, flink-table didn't pack-in com.fasterxml.jackson.*
and the project would throw class not found exception.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/flink/shaded/calcite/com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.flink.table.explain.PlanJsonParser.getSqlExecutionPlan(PlanJsonParser.java:32)
at
org.apache.flink.table.api.BatchTableEnvironment.explain(BatchTableEnvironment.scala:143)
at
org.apache.flink.table.api.BatchTableEnvironment.explain(BatchTableEnvironment.scala:164)
at
org.apache.flink.quickstart.WordCountWithTable$.main(WordCountWithTable.scala:34)
at
org.apache.flink.quickstart.WordCountWithTable.main(WordCountWithTable.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.shaded.calcite.com.fasterxml.jackson.databind.ObjectMapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
