Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
+1 for this. We bring many SQL/API features and enhance stability in 1.10 release, and almost all of them happens in Blink planner. SQL CLI is the most convenient entrypoint for me, I believe many users will have a better experience If we set Blink planner as default planner. Best, Leonard > 在 2020年1月3日，15:16，Terry Wang 写道： > > Since what blink planner can do is a superset of flink planner, big +1 for > changing the default planner to Blink planner from my side. > > Best, > Terry Wang > > > >> 2020年1月3日 15:00，Jark Wu 写道： >> >> Hi everyone, >> >> In 1.10 release, Flink SQL supports many awesome features and improvements, >> including: >> - support watermark statement and computed column in DDL >> - fully support all data types in Hive >> - Batch SQL performance improvements (TPC-DS 7x than Hive MR) >> - support INSERT OVERWRITE and INSERT PARTITION >> >> However, all the features and improvements are only avaiable in Blink >> planner, not in Old planner. >> There are also some other features are limited in Blink planner, e.g. >> Dimension Table Join [1], >> TopN [2], Deduplicate [3], streaming aggregates optimization [4], and so on. >> >> But Old planner is still the default planner in Table API & SQL. It is >> frustrating for users to set >> to blink planner manually when every time start a SQL CLI. And it's >> surprising to see unsupported >> exception if they trying out the new features but not switch planner. >> >> SQL CLI is a very important entrypoint for trying out new feautures and >> prototyping for users. >> In order to give new planner more exposures, I would like to suggest to set >> default planner >> for SQL Client to Blink planner before 1.10 release. >> >> The approach is just changing the default SQL CLI yaml configuration[5]. In >> this way, the existing >> environment is still compatible and unaffected. >> >> Changing the default planner for the whole Table API & SQL is another topic >> and is out of scope of this discussion. >> >> What do you think? >> >> Best, >> Jark >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table >> [2]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n >> [3]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication >> [4]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html >> [5]: >> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100 >
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
Hi Jark, +1 for default blink planner in SQL-CLI. I believe this new planner can be put into practice in production. We've worked hard for nearly a year, but the old planner didn't move on. And I'd like to cc to u...@flink.apache.org. If anyone finds that blink planner has any significant defects and has a larger regression than the old planner, please let us know. We will be very grateful. Best, Jingsong Lee On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: > +1 for this. > We bring many SQL/API features and enhance stability in 1.10 release, and > almost all of them happens in Blink planner. > SQL CLI is the most convenient entrypoint for me, I believe many users > will have a better experience If we set Blink planner as default planner. > > Best, > Leonard > > > 在 2020年1月3日，15:16，Terry Wang 写道： > > > > Since what blink planner can do is a superset of flink planner, big +1 > for changing the default planner to Blink planner from my side. > > > > Best, > > Terry Wang > > > > > > > >> 2020年1月3日 15:00，Jark Wu 写道： > >> > >> Hi everyone, > >> > >> In 1.10 release, Flink SQL supports many awesome features and > improvements, > >> including: > >> - support watermark statement and computed column in DDL > >> - fully support all data types in Hive > >> - Batch SQL performance improvements (TPC-DS 7x than Hive MR) > >> - support INSERT OVERWRITE and INSERT PARTITION > >> > >> However, all the features and improvements are only avaiable in Blink > >> planner, not in Old planner. > >> There are also some other features are limited in Blink planner, e.g. > >> Dimension Table Join [1], > >> TopN [2], Deduplicate [3], streaming aggregates optimization [4], and > so on. > >> > >> But Old planner is still the default planner in Table API & SQL. It is > >> frustrating for users to set > >> to blink planner manually when every time start a SQL CLI. And it's > >> surprising to see unsupported > >> exception if they trying out the new features but not switch planner. > >> > >> SQL CLI is a very important entrypoint for trying out new feautures and > >> prototyping for users. > >> In order to give new planner more exposures, I would like to suggest to > set > >> default planner > >> for SQL Client to Blink planner before 1.10 release. > >> > >> The approach is just changing the default SQL CLI yaml > configuration[5]. In > >> this way, the existing > >> environment is still compatible and unaffected. > >> > >> Changing the default planner for the whole Table API & SQL is another > topic > >> and is out of scope of this discussion. > >> > >> What do you think? > >> > >> Best, > >> Jark > >> > >> [1]: > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table > >> [2]: > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n > >> [3]: > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication > >> [4]: > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html > >> [5]: > >> > https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100 > > > > -- Best, Jingsong Lee
[jira] [Created] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
ming li created FLINK-15467:
---
Summary: Should wait for the end of the source thread during the
Task cancellation
Key: FLINK-15467
URL: https://issues.apache.org/jira/browse/FLINK-15467
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.9.1, 1.9.0
Reporter: ming li
In the new mailBox model, SourceStreamTask starts a source thread to run
user methods, and the current execution thread will block on mailbox.takeMail
(). When a task cancels, the TaskCanceler thread will cancel the task and
interrupt the execution thread. Therefore, the execution thread of
SourceStreamTask will throw InterruptedException, then cancel the task again,
and throw an exception.
{code:java}
//代码占位符
@Override
protected void performDefaultAction(ActionContext context) throws Exception {
// Against the usual contract of this method, this implementation is not
step-wise but blocking instead for
// compatibility reasons with the current source interface (source functions
run as a loop, not in steps).
sourceThread.start();
// We run an alternative mailbox loop that does not involve default actions
and synchronizes around actions.
try {
runAlternativeMailboxLoop();
} catch (Exception mailboxEx) {
// We cancel the source function if some runtime exception escaped the
mailbox.
if (!isCanceled()) {
cancelTask();
}
throw mailboxEx;
}
sourceThread.join();
if (!isFinished) {
sourceThread.checkThrowSourceExecutionException();
}
context.allActionsCompleted();
}
{code}
When all tasks of this TaskExecutor are canceled, the blob file will be
cleaned up. But the real source thread is not finished at this time, which will
cause a ClassNotFoundException when loading a new class. In this case, the
source thread may not be able to properly clean up and release resources (such
as closing child threads, cleaning up local files, etc.). Therefore, I think we
should mark this task canceled or finished after the execution of the source
thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15468) INSERT OVERWRITE not supported from SQL CLI
Rui Li created FLINK-15468: -- Summary: INSERT OVERWRITE not supported from SQL CLI Key: FLINK-15468 URL: https://issues.apache.org/jira/browse/FLINK-15468 Project: Flink Issue Type: Bug Components: Table SQL / Client Reporter: Rui Li Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
+1, I have already made blink as the default planner of flink interpreter in Zeppelin Jingsong Li 于2020年1月3日周五 下午4:37写道： > Hi Jark, > > +1 for default blink planner in SQL-CLI. > I believe this new planner can be put into practice in production. > We've worked hard for nearly a year, but the old planner didn't move on. > > And I'd like to cc to u...@flink.apache.org. > If anyone finds that blink planner has any significant defects and has a > larger regression than the old planner, please let us know. We will be very > grateful. > > Best, > Jingsong Lee > > On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: > >> +1 for this. >> We bring many SQL/API features and enhance stability in 1.10 release, and >> almost all of them happens in Blink planner. >> SQL CLI is the most convenient entrypoint for me, I believe many users >> will have a better experience If we set Blink planner as default planner. >> >> Best, >> Leonard >> >> > 在 2020年1月3日，15:16，Terry Wang 写道： >> > >> > Since what blink planner can do is a superset of flink planner, big +1 >> for changing the default planner to Blink planner from my side. >> > >> > Best, >> > Terry Wang >> > >> > >> > >> >> 2020年1月3日 15:00，Jark Wu 写道： >> >> >> >> Hi everyone, >> >> >> >> In 1.10 release, Flink SQL supports many awesome features and >> improvements, >> >> including: >> >> - support watermark statement and computed column in DDL >> >> - fully support all data types in Hive >> >> - Batch SQL performance improvements (TPC-DS 7x than Hive MR) >> >> - support INSERT OVERWRITE and INSERT PARTITION >> >> >> >> However, all the features and improvements are only avaiable in Blink >> >> planner, not in Old planner. >> >> There are also some other features are limited in Blink planner, e.g. >> >> Dimension Table Join [1], >> >> TopN [2], Deduplicate [3], streaming aggregates optimization [4], and >> so on. >> >> >> >> But Old planner is still the default planner in Table API & SQL. It is >> >> frustrating for users to set >> >> to blink planner manually when every time start a SQL CLI. And it's >> >> surprising to see unsupported >> >> exception if they trying out the new features but not switch planner. >> >> >> >> SQL CLI is a very important entrypoint for trying out new feautures and >> >> prototyping for users. >> >> In order to give new planner more exposures, I would like to suggest >> to set >> >> default planner >> >> for SQL Client to Blink planner before 1.10 release. >> >> >> >> The approach is just changing the default SQL CLI yaml >> configuration[5]. In >> >> this way, the existing >> >> environment is still compatible and unaffected. >> >> >> >> Changing the default planner for the whole Table API & SQL is another >> topic >> >> and is out of scope of this discussion. >> >> >> >> What do you think? >> >> >> >> Best, >> >> Jark >> >> >> >> [1]: >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table >> >> [2]: >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n >> >> [3]: >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication >> >> [4]: >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html >> >> [5]: >> >> >> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100 >> > >> >> > > -- > Best, Jingsong Lee > -- Best Regards Jeff Zhang
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
Hi Jark, +1 for making the Blink planner the default planner for the SQL Client. I think for the Table API, we should give the planner a bit more exposure and target the changing of a default planner for 1.11. What do you think about this two step approach? Regards, Timo On 03.01.20 09:37, Jingsong Li wrote: Hi Jark, +1 for default blink planner in SQL-CLI. I believe this new planner can be put into practice in production. We've worked hard for nearly a year, but the old planner didn't move on. And I'd like to cc to u...@flink.apache.org. If anyone finds that blink planner has any significant defects and has a larger regression than the old planner, please let us know. We will be very grateful. Best, Jingsong Lee On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: +1 for this. We bring many SQL/API features and enhance stability in 1.10 release, and almost all of them happens in Blink planner. SQL CLI is the most convenient entrypoint for me, I believe many users will have a better experience If we set Blink planner as default planner. Best, Leonard 在 2020年1月3日，15:16，Terry Wang 写道： Since what blink planner can do is a superset of flink planner, big +1 for changing the default planner to Blink planner from my side. Best, Terry Wang 2020年1月3日 15:00，Jark Wu 写道： Hi everyone, In 1.10 release, Flink SQL supports many awesome features and improvements, including: - support watermark statement and computed column in DDL - fully support all data types in Hive - Batch SQL performance improvements (TPC-DS 7x than Hive MR) - support INSERT OVERWRITE and INSERT PARTITION However, all the features and improvements are only avaiable in Blink planner, not in Old planner. There are also some other features are limited in Blink planner, e.g. Dimension Table Join [1], TopN [2], Deduplicate [3], streaming aggregates optimization [4], and so on. But Old planner is still the default planner in Table API & SQL. It is frustrating for users to set to blink planner manually when every time start a SQL CLI. And it's surprising to see unsupported exception if they trying out the new features but not switch planner. SQL CLI is a very important entrypoint for trying out new feautures and prototyping for users. In order to give new planner more exposures, I would like to suggest to set default planner for SQL Client to Blink planner before 1.10 release. The approach is just changing the default SQL CLI yaml configuration[5]. In this way, the existing environment is still compatible and unaffected. Changing the default planner for the whole Table API & SQL is another topic and is out of scope of this discussion. What do you think? Best, Jark [1]: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table [2]: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n [3]: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication [4]: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html [5]: https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
Hi Timo, I have the same idea about the two step approach. +1 to change default planner for Table API in 1.11. Best, Jark On Fri, 3 Jan 2020 at 17:06, Timo Walther wrote: > Hi Jark, > > +1 for making the Blink planner the default planner for the SQL Client. > > I think for the Table API, we should give the planner a bit more > exposure and target the changing of a default planner for 1.11. > > What do you think about this two step approach? > > Regards, > Timo > > On 03.01.20 09:37, Jingsong Li wrote: > > Hi Jark, > > > > +1 for default blink planner in SQL-CLI. > > I believe this new planner can be put into practice in production. > > We've worked hard for nearly a year, but the old planner didn't move on. > > > > And I'd like to cc to u...@flink.apache.org. > > If anyone finds that blink planner has any significant defects and has a > > larger regression than the old planner, please let us know. We will be > very > > grateful. > > > > Best, > > Jingsong Lee > > > > On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: > > > >> +1 for this. > >> We bring many SQL/API features and enhance stability in 1.10 release, > and > >> almost all of them happens in Blink planner. > >> SQL CLI is the most convenient entrypoint for me, I believe many users > >> will have a better experience If we set Blink planner as default > planner. > >> > >> Best, > >> Leonard > >> > >>> 在 2020年1月3日，15:16，Terry Wang 写道： > >>> > >>> Since what blink planner can do is a superset of flink planner, big +1 > >> for changing the default planner to Blink planner from my side. > >>> > >>> Best, > >>> Terry Wang > >>> > >>> > >>> > 2020年1月3日 15:00，Jark Wu 写道： > > Hi everyone, > > In 1.10 release, Flink SQL supports many awesome features and > >> improvements, > including: > - support watermark statement and computed column in DDL > - fully support all data types in Hive > - Batch SQL performance improvements (TPC-DS 7x than Hive MR) > - support INSERT OVERWRITE and INSERT PARTITION > > However, all the features and improvements are only avaiable in Blink > planner, not in Old planner. > There are also some other features are limited in Blink planner, e.g. > Dimension Table Join [1], > TopN [2], Deduplicate [3], streaming aggregates optimization [4], and > >> so on. > > But Old planner is still the default planner in Table API & SQL. It is > frustrating for users to set > to blink planner manually when every time start a SQL CLI. And it's > surprising to see unsupported > exception if they trying out the new features but not switch planner. > > SQL CLI is a very important entrypoint for trying out new feautures > and > prototyping for users. > In order to give new planner more exposures, I would like to suggest > to > >> set > default planner > for SQL Client to Blink planner before 1.10 release. > > The approach is just changing the default SQL CLI yaml > >> configuration[5]. In > this way, the existing > environment is still compatible and unaffected. > > Changing the default planner for the whole Table API & SQL is another > >> topic > and is out of scope of this discussion. > > What do you think? > > Best, > Jark > > [1]: > > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table > [2]: > > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n > [3]: > > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication > [4]: > > >> > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html > [5]: > > >> > https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100 > >>> > >> > >> > > > >
Re: [DISCUSS] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL
Hi,
sorry for jumping into the discussion so late. I had a quick look at the
FLIP. It looks very nice and detailed. I have one question that I could
not find in the FLIP itself. Maybe it is hidden in the long discussion
thread.
What are the return types of all functions? Do we introduce new types
with this FLIP? Also the RAW types should be avoided. Do all functions
return STRING/BOOLEAN?
Thanks,
Timo
On 31.12.19 09:39, Hequn Cheng wrote:
Thanks a lot for the update. +1 to start a vote.
On Tue, Dec 31, 2019 at 2:38 PM Forward Xu wrote:
Hi Jark, Hequn,
I have updated the documentation.
Best,
Forward
Forward Xu 于2019年12月29日周日 下午4:01写道：
Hi Jark, Hequn,
Thank you very much, Introducing new `TableSymbol`s sounds like a good
idea. +1 for the proposal.
I think this idea is good, I will add this in the documentation.
Best, Forward
Hequn Cheng 于2019年12月29日周日 下午3:41写道：
Hi Jark,
Introducing new `TableSymbol`s sounds like a good idea. +1 for the
proposal.
@ForwardXu what do you think? Would be great if the document can be
updated
accordingly.
Best, Hequn
On Fri, Dec 27, 2019 at 4:14 PM Jark Wu wrote:
Thanks for looking into the design Hequn. I agree it would be great to
have a full story design.
For the ON ERROR and ON EMPTY clause in Table API, some initial
thoughts in my mind is that
we can introduce some new `TableSymbol`s as the second parameter of
json
function, e.g. `JsonErrorStrategy`.
For example,
JSON_VALUE(v, 'lax $.b' ERROR ON ERROR)
== is equal to Table API ==>
v.jsonValue("lax $.b", JsonErrorStrategy.ERROR)
Best,
Jark
On Thu, 26 Dec 2019 at 23:37, Hequn Cheng
wrote:
Hi Jark & ForwardXu,
The design doc looks very nice! Only some minor feedback from my
side.
As calcite has already implemented the JSON functions, I would
suppose
the semantics and implementation are right for SQL.
For TableAPI, I think the most important is to keep align with the
SQL(which has also been mentioned by Jark in the previous
discussion).
Have
an equivalent feature set for all APIs and maintain it otherwise
confusion
increases especially when more and more functions are added. The
document
has documented how to support TableAPI. I think this is very good!
And
it
would be better to also include ON ERROR or ON EMPTY for Table API.
We
can
implement these features step by step, but maybe we should design all
these
once for all to avoid API changes later. Meanwhile, these features
are
also
commonly required by users.
Would be great to also have your opinions!
Best,
Hequn
On Mon, Dec 23, 2019 at 10:15 AM Jark Wu wrote:
Hi Forward,
Thanks for creating the FLIP. +1 to start a vote.
@Hequn Cheng @Kurt Young
,
could you help to review the design doc too?
Best,
Jark
On Mon, 23 Dec 2019 at 10:10, tison wrote:
modified:
https://lists.apache.org/x/thread.html/b3c0265cc2b660fe11ce550b84a831a7606de12908ff7ff0959a4794@%3Cdev.flink.apache.org%3E
Re: [DISCUSS] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL
Hi Timo,
That's a good point.
We didn't introduce any new types. We will use the function definition
defined by Calcite[1].
So all the functions return STRING/BOOLEAN.
Hi Forward,
I think we may need an additional column to describe the return type of
each function.
Best,
Jark
[1]:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlJsonQueryFunction.java
On Fri, 3 Jan 2020 at 17:30, Timo Walther wrote:
> Hi,
>
> sorry for jumping into the discussion so late. I had a quick look at the
> FLIP. It looks very nice and detailed. I have one question that I could
> not find in the FLIP itself. Maybe it is hidden in the long discussion
> thread.
>
> What are the return types of all functions? Do we introduce new types
> with this FLIP? Also the RAW types should be avoided. Do all functions
> return STRING/BOOLEAN?
>
> Thanks,
> Timo
>
>
> On 31.12.19 09:39, Hequn Cheng wrote:
> > Thanks a lot for the update. +1 to start a vote.
> >
> > On Tue, Dec 31, 2019 at 2:38 PM Forward Xu
> wrote:
> >
> >> Hi Jark, Hequn,
> >>
> >> I have updated the documentation.
> >>
> >> Best,
> >>
> >> Forward
> >>
> >> Forward Xu 于2019年12月29日周日 下午4:01写道：
> >>
> >>> Hi Jark, Hequn,
> >>>
> >>> Thank you very much, Introducing new `TableSymbol`s sounds like a good
> >>> idea. +1 for the proposal.
> >>>
> >>> I think this idea is good, I will add this in the documentation.
> >>>
> >>>
> >>> Best, Forward
> >>>
> >>> Hequn Cheng 于2019年12月29日周日 下午3:41写道：
> >>>
> Hi Jark,
>
> Introducing new `TableSymbol`s sounds like a good idea. +1 for the
> proposal.
> @ForwardXu what do you think? Would be great if the document can be
> updated
> accordingly.
>
> Best, Hequn
>
>
> On Fri, Dec 27, 2019 at 4:14 PM Jark Wu wrote:
>
> > Thanks for looking into the design Hequn. I agree it would be great
> to
> > have a full story design.
> >
> > For the ON ERROR and ON EMPTY clause in Table API, some initial
> > thoughts in my mind is that
> > we can introduce some new `TableSymbol`s as the second parameter of
> >> json
> > function, e.g. `JsonErrorStrategy`.
> >
> > For example,
> >
> > JSON_VALUE(v, 'lax $.b' ERROR ON ERROR)
> > == is equal to Table API ==>
> > v.jsonValue("lax $.b", JsonErrorStrategy.ERROR)
> >
> > Best,
> > Jark
> >
> >
> > On Thu, 26 Dec 2019 at 23:37, Hequn Cheng
> >> wrote:
> >
> >> Hi Jark & ForwardXu,
> >>
> >> The design doc looks very nice! Only some minor feedback from my
> >> side.
> >>
> >> As calcite has already implemented the JSON functions, I would
> >> suppose
> >> the semantics and implementation are right for SQL.
> >>
> >> For TableAPI, I think the most important is to keep align with the
> >> SQL(which has also been mentioned by Jark in the previous
> >> discussion).
> Have
> >> an equivalent feature set for all APIs and maintain it otherwise
> confusion
> >> increases especially when more and more functions are added. The
> document
> >> has documented how to support TableAPI. I think this is very good!
> >> And
> it
> >> would be better to also include ON ERROR or ON EMPTY for Table API.
> >> We
> can
> >> implement these features step by step, but maybe we should design
> all
> these
> >> once for all to avoid API changes later. Meanwhile, these features
> >> are
> also
> >> commonly required by users.
> >>
> >> Would be great to also have your opinions!
> >>
> >> Best,
> >> Hequn
> >>
> >>
> >> On Mon, Dec 23, 2019 at 10:15 AM Jark Wu wrote:
> >>
> >>> Hi Forward,
> >>>
> >>> Thanks for creating the FLIP. +1 to start a vote.
> >>>
> >>> @Hequn Cheng @Kurt Young <
> ykt...@gmail.com>
> >> ,
> >>> could you help to review the design doc too?
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>>
> >>> On Mon, 23 Dec 2019 at 10:10, tison wrote:
> >>>
> modified:
>
>
>
> >>
> https://lists.apache.org/x/thread.html/b3c0265cc2b660fe11ce550b84a831a7606de12908ff7ff0959a4794@%3Cdev.flink.apache.org%3E
>
> >>>
>
> >>>
> >>
> >
>
>
[jira] [Created] (FLINK-15469) UpsertStreamTableSink should support new type system
Zhenghua Gao created FLINK-15469:
Summary: UpsertStreamTableSink should support new type system
Key: FLINK-15469
URL: https://issues.apache.org/jira/browse/FLINK-15469
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.11.0
Currently *UpsertStreamTableSink* can only returns TypeInformation of the
requested record, which can't support types with precision and scale, e.g.
TIMESTAMP(p), DECIMAL(p,s).
A proposal is deprecating the *getRecordType* API and adding a
*getRecordDataType* API instead to return the data type of the requested record.
{code:java}
/**
* Returns the requested record type.
*
* @Deprecated This method will be removed in future versions. It's recommended
to use {@link #getRecordDataType()} instead.
*/
@Deprecated
TypeInformation getRecordType();
/*
* Returns the requested record data type.
*/
DataType getRecordDataType();
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #3
Thanks for the great work, and look forward to the first RC. Best, Congxian Jingsong Li 于2020年1月3日周五 下午12:58写道： > Thanks Yu and Gary, > > Table and hive layer have resolved all blockers and critical issues. (38 > blockers and 33 critical issues) > > Look forward to the first RC. > > Best, > Jingsong Lee > > On Fri, Jan 3, 2020 at 11:16 AM Danny Chan wrote: > > > Thanks for the great work ~ > > > > Best, > > Danny Chan > > 在 2020年1月2日 +0800 PM8:48，Yu Li ，写道： > > > Hi community, > > > > > > It has been three weeks since we cut the release-1.10 branch and kicked > > off > > > the release testing phase [1]. Thanks to all the efforts of the > > community, > > > 46 blocker and 11 critical issues [2] have been resolved during this > > > period, and now we are left with only 12 blocker issues [3]. > > > > > > If everything goes smoothly, we could possibly resolve all blockers and > > > produce the first RC in the 3rd week of Jan. Please be prepared for > > testing > > > the RC and let us know if there are any questions. > > > > > > Best, > > > Gary and Yu > > > > > > [1] https://s.apache.org/1okuv > > > [2] https://s.apache.org/gbd0r > > > [3] https://s.apache.org/euzaj > > > > > -- > Best, Jingsong Lee >
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #3
Hi Yu, Thanks for your great work. Best, Vino Congxian Qiu 于2020年1月3日周五 下午6:18写道： > Thanks for the great work, and look forward to the first RC. > > Best, > Congxian > > > Jingsong Li 于2020年1月3日周五 下午12:58写道： > > > Thanks Yu and Gary, > > > > Table and hive layer have resolved all blockers and critical issues. (38 > > blockers and 33 critical issues) > > > > Look forward to the first RC. > > > > Best, > > Jingsong Lee > > > > On Fri, Jan 3, 2020 at 11:16 AM Danny Chan wrote: > > > > > Thanks for the great work ~ > > > > > > Best, > > > Danny Chan > > > 在 2020年1月2日 +0800 PM8:48，Yu Li ，写道： > > > > Hi community, > > > > > > > > It has been three weeks since we cut the release-1.10 branch and > kicked > > > off > > > > the release testing phase [1]. Thanks to all the efforts of the > > > community, > > > > 46 blocker and 11 critical issues [2] have been resolved during this > > > > period, and now we are left with only 12 blocker issues [3]. > > > > > > > > If everything goes smoothly, we could possibly resolve all blockers > and > > > > produce the first RC in the 3rd week of Jan. Please be prepared for > > > testing > > > > the RC and let us know if there are any questions. > > > > > > > > Best, > > > > Gary and Yu > > > > > > > > [1] https://s.apache.org/1okuv > > > > [2] https://s.apache.org/gbd0r > > > > [3] https://s.apache.org/euzaj > > > > > > > > > -- > > Best, Jingsong Lee > > >
Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release
Hi Benoît, Thanks for the reminder. I will look into the issue and hopefully we can target it into 1.9.2 and 1.10. Cheers, Jark On Fri, 3 Jan 2020 at 18:21, Benoît Paris < benoit.pa...@centraliens-lille.org> wrote: > > If anyone finds that blink planner has any significant defects and has > a larger regression than the old planner, please let us know. > > Overall, the Blink-exclusive features are must (TopN, deduplicate, > LAST_VALUE, plan reuse, etc)! But all use cases of the Legacy planner in > production are not covered: > An edge case of Temporal Table Functions does not allow computed Tables > (as opposed to TableSources) to be used on the query side in Blink ( > https://issues.apache.org/jira/browse/FLINK-14200) > > Cheers > Ben > > > On Fri, Jan 3, 2020 at 10:00 AM Jeff Zhang wrote: > >> +1, I have already made blink as the default planner of flink interpreter >> in Zeppelin >> >> >> Jingsong Li 于2020年1月3日周五 下午4:37写道： >> >>> Hi Jark, >>> >>> +1 for default blink planner in SQL-CLI. >>> I believe this new planner can be put into practice in production. >>> We've worked hard for nearly a year, but the old planner didn't move on. >>> >>> And I'd like to cc to u...@flink.apache.org. >>> If anyone finds that blink planner has any significant defects and has a >>> larger regression than the old planner, please let us know. We will be very >>> grateful. >>> >>> Best, >>> Jingsong Lee >>> >>> On Fri, Jan 3, 2020 at 4:14 PM Leonard Xu wrote: >>> +1 for this. We bring many SQL/API features and enhance stability in 1.10 release, and almost all of them happens in Blink planner. SQL CLI is the most convenient entrypoint for me, I believe many users will have a better experience If we set Blink planner as default planner. Best, Leonard > 在 2020年1月3日，15:16，Terry Wang 写道： > > Since what blink planner can do is a superset of flink planner, big +1 for changing the default planner to Blink planner from my side. > > Best, > Terry Wang > > > >> 2020年1月3日 15:00，Jark Wu 写道： >> >> Hi everyone, >> >> In 1.10 release, Flink SQL supports many awesome features and improvements, >> including: >> - support watermark statement and computed column in DDL >> - fully support all data types in Hive >> - Batch SQL performance improvements (TPC-DS 7x than Hive MR) >> - support INSERT OVERWRITE and INSERT PARTITION >> >> However, all the features and improvements are only avaiable in Blink >> planner, not in Old planner. >> There are also some other features are limited in Blink planner, e.g. >> Dimension Table Join [1], >> TopN [2], Deduplicate [3], streaming aggregates optimization [4], and so on. >> >> But Old planner is still the default planner in Table API & SQL. It is >> frustrating for users to set >> to blink planner manually when every time start a SQL CLI. And it's >> surprising to see unsupported >> exception if they trying out the new features but not switch planner. >> >> SQL CLI is a very important entrypoint for trying out new feautures and >> prototyping for users. >> In order to give new planner more exposures, I would like to suggest to set >> default planner >> for SQL Client to Blink planner before 1.10 release. >> >> The approach is just changing the default SQL CLI yaml configuration[5]. In >> this way, the existing >> environment is still compatible and unaffected. >> >> Changing the default planner for the whole Table API & SQL is another topic >> and is out of scope of this discussion. >> >> What do you think? >> >> Best, >> Jark >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table >> [2]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#top-n >> [3]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication >> [4]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tuning/streaming_aggregation_optimization.html >> [5]: >> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/conf/sql-client-defaults.yaml#L100 > >>> >>> -- >>> Best, Jingsong Lee >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > -- > Benoît Paris > Ingénieur Machine Learning Explicable > Tél : +33 6 60 74 23 00 > http://benoit.paris > http://explicable.ml >
[jira] [Created] (FLINK-15470) Remove YARN magic properties file
Yang Wang created FLINK-15470:
-
Summary: Remove YARN magic properties file
Key: FLINK-15470
URL: https://issues.apache.org/jira/browse/FLINK-15470
Project: Flink
Issue Type: Task
Components: Command Line Client, Deployment / YARN
Reporter: Yang Wang
Currently, when we start a session, flink client will write a yarn properties
under \{java.io.tmpdir}/.yarn-properties-\{user-name}. So the user could use a
simple command {{./bin/flink run ./examples/batch/WordCount.jar}} to submit a
job to the session.
Even if the session cluster has finished or kill by {{yarn application -kill}},
it will still submit a job to the non-existing session. Some users has run into
this problem in the ML. So i suggest to remove the properties file and always
use {{-yid application_id}} to submit job an existing session.
[https://lists.apache.org/thread.html/b0e0e388a16d1891d1de651bec027a3820130f28aaaeb843440cb679%40%3Cuser.flink.apache.org%3E]
We also have some discussion under
[FLINK-15179|https://issues.apache.org/jira/browse/FLINK-15179]. The cli
options will be deprecated in the future and we hope {{FlinkYarnSessionCli}}
could use {{ExecutorCli}}. So we will try to clean up the code, i think the
magic properties file is the case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15471) HA e2e check for empty .out files isn't working
Chesnay Schepler created FLINK-15471:
Summary: HA e2e check for empty .out files isn't working
Key: FLINK-15471
URL: https://issues.apache.org/jira/browse/FLINK-15471
Project: Flink
Issue Type: Improvement
Components: Test Infrastructure
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.9.2, 1.10.0
{{common_ha.sh#verify_logs:}}
{code}
if ! check_logs_for_non_empty_out_files; then
echo "FAILURE: Alerts found at the general purpose job."
EXIT_CODE=1
fi
{code}
Since check_logs_for_non_empty_out_files does only set EXIT_CODE without
modifying the return value the check will never fail.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15472) Support SQL Client Gateway
godfrey he created FLINK-15472: -- Summary: Support SQL Client Gateway Key: FLINK-15472 URL: https://issues.apache.org/jira/browse/FLINK-15472 Project: Flink Issue Type: New Feature Components: Table SQL / Client Reporter: godfrey he FLIP-91: https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway design document: https://docs.google.com/document/d/1T7--664rts4t_4gjRPw937S9ln9Plf1yghNQ9IiHQtQ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15473) Add support for Linux on ppc64le to MemoryArchitecture
Ronald O. Edmark created FLINK-15473:
Summary: Add support for Linux on ppc64le to MemoryArchitecture
Key: FLINK-15473
URL: https://issues.apache.org/jira/browse/FLINK-15473
Project: Flink
Issue Type: Improvement
Environment: Red Hat, Ubuntu Linux on PowerPC.
Reporter: Ronald O. Edmark
Similar to issue FLINK-13449
Please add "ppc64le" to MemoryArchitecture.java
final List names64bit = Arrays.asList("amd64", "x86_64", "aarch64",
"ppc64le");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL
Hi Timo, Jack,
Well, I added additional column to describe the return type of each
function and
updated the google doc.
Best,
Forward
Jark Wu 于2020年1月3日周五 下午5:48写道：
> Hi Timo,
>
> That's a good point.
> We didn't introduce any new types. We will use the function definition
> defined by Calcite[1].
> So all the functions return STRING/BOOLEAN.
>
> Hi Forward,
> I think we may need an additional column to describe the return type of
> each function.
>
> Best,
> Jark
>
> [1]:
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlJsonQueryFunction.java
>
> On Fri, 3 Jan 2020 at 17:30, Timo Walther wrote:
>
> > Hi,
> >
> > sorry for jumping into the discussion so late. I had a quick look at the
> > FLIP. It looks very nice and detailed. I have one question that I could
> > not find in the FLIP itself. Maybe it is hidden in the long discussion
> > thread.
> >
> > What are the return types of all functions? Do we introduce new types
> > with this FLIP? Also the RAW types should be avoided. Do all functions
> > return STRING/BOOLEAN?
> >
> > Thanks,
> > Timo
> >
> >
> > On 31.12.19 09:39, Hequn Cheng wrote:
> > > Thanks a lot for the update. +1 to start a vote.
> > >
> > > On Tue, Dec 31, 2019 at 2:38 PM Forward Xu
> > wrote:
> > >
> > >> Hi Jark, Hequn,
> > >>
> > >> I have updated the documentation.
> > >>
> > >> Best,
> > >>
> > >> Forward
> > >>
> > >> Forward Xu 于2019年12月29日周日 下午4:01写道：
> > >>
> > >>> Hi Jark, Hequn,
> > >>>
> > >>> Thank you very much, Introducing new `TableSymbol`s sounds like a
> good
> > >>> idea. +1 for the proposal.
> > >>>
> > >>> I think this idea is good, I will add this in the documentation.
> > >>>
> > >>>
> > >>> Best, Forward
> > >>>
> > >>> Hequn Cheng 于2019年12月29日周日 下午3:41写道：
> > >>>
> > Hi Jark,
> >
> > Introducing new `TableSymbol`s sounds like a good idea. +1 for the
> > proposal.
> > @ForwardXu what do you think? Would be great if the document can be
> > updated
> > accordingly.
> >
> > Best, Hequn
> >
> >
> > On Fri, Dec 27, 2019 at 4:14 PM Jark Wu wrote:
> >
> > > Thanks for looking into the design Hequn. I agree it would be great
> > to
> > > have a full story design.
> > >
> > > For the ON ERROR and ON EMPTY clause in Table API, some initial
> > > thoughts in my mind is that
> > > we can introduce some new `TableSymbol`s as the second parameter of
> > >> json
> > > function, e.g. `JsonErrorStrategy`.
> > >
> > > For example,
> > >
> > > JSON_VALUE(v, 'lax $.b' ERROR ON ERROR)
> > > == is equal to Table API ==>
> > > v.jsonValue("lax $.b", JsonErrorStrategy.ERROR)
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > On Thu, 26 Dec 2019 at 23:37, Hequn Cheng
> > >> wrote:
> > >
> > >> Hi Jark & ForwardXu,
> > >>
> > >> The design doc looks very nice! Only some minor feedback from my
> > >> side.
> > >>
> > >> As calcite has already implemented the JSON functions, I would
> > >> suppose
> > >> the semantics and implementation are right for SQL.
> > >>
> > >> For TableAPI, I think the most important is to keep align with the
> > >> SQL(which has also been mentioned by Jark in the previous
> > >> discussion).
> > Have
> > >> an equivalent feature set for all APIs and maintain it otherwise
> > confusion
> > >> increases especially when more and more functions are added. The
> > document
> > >> has documented how to support TableAPI. I think this is very good!
> > >> And
> > it
> > >> would be better to also include ON ERROR or ON EMPTY for Table
> API.
> > >> We
> > can
> > >> implement these features step by step, but maybe we should design
> > all
> > these
> > >> once for all to avoid API changes later. Meanwhile, these features
> > >> are
> > also
> > >> commonly required by users.
> > >>
> > >> Would be great to also have your opinions!
> > >>
> > >> Best,
> > >> Hequn
> > >>
> > >>
> > >> On Mon, Dec 23, 2019 at 10:15 AM Jark Wu
> wrote:
> > >>
> > >>> Hi Forward,
> > >>>
> > >>> Thanks for creating the FLIP. +1 to start a vote.
> > >>>
> > >>> @Hequn Cheng @Kurt Young <
> > ykt...@gmail.com>
> > >> ,
> > >>> could you help to review the design doc too?
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>>
> > >>> On Mon, 23 Dec 2019 at 10:10, tison
> wrote:
> > >>>
> > modified:
> >
> >
> >
> > >>
> >
> https://lists.apache.org/x/thread.html/b3c0265cc2b660fe11ce550b84a831a7606de12908ff7ff0959a4794@%3Cdev.flink.apache.org%3E
> >
> > >>>
> >
> > >>>
> > >>
> > >
> >
> >
>
[jira] [Created] (FLINK-15474) In TypeSerializerUpgradeTestBase, create serializer snapshots "on demand"
Aljoscha Krettek created FLINK-15474: Summary: In TypeSerializerUpgradeTestBase, create serializer snapshots "on demand" Key: FLINK-15474 URL: https://issues.apache.org/jira/browse/FLINK-15474 Project: Flink Issue Type: Bug Components: API / Type Serialization System, Tests Affects Versions: 1.9.0, 1.10.0 Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.10.0 Currently, we store binary snapshots in the repository for all the different serializer upgrade test configurations (see linked POC for which snapshots are there for just the POJO serializer). This is hard to maintain, because someone has to go back and generate snapshtos from previous Flink versions and add them to the repo when updating the tests for a new Flink version. It's also problematic from a repository perspective because we keep piling up binary snapshots. Instead, we can create a snapshot "on demand" from a previous Flink version by using a classloader that has the previous Flink jar. I created a POC which demonstrated the approach: [https://github.com/aljoscha/flink/tree/jit-serializer-test-base]. The advantage is that we don't need binary snapshots in the repo anymore, updating the tests to a newer Flink version should be as easy as adding a new migration version and Flink download url. The downside is that the test now downloads Flink releases (in the PoC this is done using the caching infra introduced for e2e tests), which is costly and also re-generates the snapshots for every test, which is also costly. The test time (minus downloading) goes up from about 300 ms to roughly 6 seconds. That's not something I would call a unit test. We could call these "integration tests" (or even e2e tests) and only run them nightly. Side note, we don't have test coverage for serializer upgrades from 1.8 and 1.9 currently, so even doing it nightly would be an improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15475) Add isOutputTypeUsed() API to Transformation
Rong Rong created FLINK-15475:
-
Summary: Add isOutputTypeUsed() API to Transformation
Key: FLINK-15475
URL: https://issues.apache.org/jira/browse/FLINK-15475
Project: Flink
Issue Type: Improvement
Components: API / Core, API / DataSet, API / DataStream
Reporter: Rong Rong
Assignee: Rong Rong
Currently there's no way to "peek" into a Transformation and see if OutputType
has been used or not. The only way is to invoke the {{setOutputType}} API and
wrap around it with a try-catch block.
It would be nice if we have a `isTypeUsed()` or `isOutputTypeUsed()` API to
check whether a particular transformation has a definitive output type set /
used or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15476) Update StreamingFileSink documentation -- bulk encoded writer now supports customized checkpoint policy
Ying Xu created FLINK-15476: --- Summary: Update StreamingFileSink documentation -- bulk encoded writer now supports customized checkpoint policy Key: FLINK-15476 URL: https://issues.apache.org/jira/browse/FLINK-15476 Project: Flink Issue Type: Improvement Reporter: Ying Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
