[DISCUSS] Flink SQL DDL Design

[Shuyi Chen]

Hi everyone,

SQL DDL support has been a long-time ask from the community. Current Flink
SQL support only DML (e.g. SELECT and INSERT statements). In its current
form, Flink SQL users still need to define/create table sources and sinks
programmatically in Java/Scala. Also, in SQL Client, without DDL support,
the current implementation does not allow dynamical creation of table, type
or functions with SQL, this adds friction for its adoption.

I drafted a design doc [1] with a few other community members that proposes
the design and implementation for adding DDL support in Flink. The initial
design considers DDL for table, view, type, library and function. It will
be great to get feedback on the design from the community, and align with
latest effort in unified SQL connector API  [2] and Flink Hive integration
[3].

Any feedback is highly appreciated.

Thanks
Shuyi Chen

[1]
https://docs.google.com/document/d/1TTP-GCC8wSsibJaSUyFZ_5NBAHYEB1FVmPpP7RgDGBA/edit?usp=sharing
[2]
https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing
[3]
https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
-- 
"So you have to trust that the dots will somehow connect in your future."

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Tzu-Li Chen]

Hi jingchengm

Thanks a lot for your proposal! I find it is a good start point for
internal optimization works and help Flink to be more
user-friendly.

AFAIK, DataStream is the most popular API currently that Flink
users should describe their logic with detailed logic.
>From a more internal view the conversion from DataStream to
JobGraph is quite mechanically and hard to be optimized. So when
users program with DataStream, they have to learn more internals
and spend a lot of time to tune for performance.
With your proposal, we provide enhanced functionality of Table API,
so that users can describe their job easily on Table aspect. This gives
an opportunity to Flink developers to introduce an optimize phase
while transforming user program(described by Table API) to internal
representation.

Given a user who want to start using Flink with simple ETL, pipelining
or analytics, he would find it is most naturally described by SQL/Table
API. Further, as mentioned by @hequn,

SQL is a widely used language. It follows standards, is a
> descriptive language, and is easy to use


thus we could expect with the enhancement of SQL/Table API, Flink
becomes more friendly to users.

Looking forward to the design doc/FLIP!

Best,
tison.


jincheng sun  于2018年11月2日周五 上午11:46写道：

> Hi Hequn,
> Thanks for your feedback! And also thanks for our offline discussion!
> You are right, unification of batch and streaming is very important for
> flink API.
> We will provide more detailed design later, Please let me know if you have
> further thoughts or feedback.
>
> Thanks,
> Jincheng
>
> Hequn Cheng  于2018年11月2日周五 上午10:02写道：
>
> > Hi Jincheng,
> >
> > Thanks a lot for your proposal. It is very encouraging!
> >
> > As we all know, SQL is a widely used language. It follows standards, is a
> > descriptive language, and is easy to use. A powerful feature of SQL is
> that
> > it supports optimization. Users only need to care about the logic of the
> > program. The underlying optimizer will help users optimize the
> performance
> > of the program. However, in terms of functionality and ease of use, in
> some
> > scenarios sql will be limited, as described in Jincheng's proposal.
> >
> > Correspondingly, the DataStream/DataSet api can provide powerful
> > functionalities. Users can write ProcessFunction/CoProcessFunction and
> get
> > the timer. Compared with SQL, it provides more functionalities and
> > flexibilities. However, it does not support optimization like SQL.
> > Meanwhile, DataStream/DataSet api has not been unified which means, for
> the
> > same logic, users need to write a job for each stream and batch.
> >
> > With TableApi, I think we can combine the advantages of both. Users can
> > easily write relational operations and enjoy optimization. At the same
> > time, it supports more functionality and ease of use. Looking forward to
> > the detailed design/FLIP.
> >
> > Best,
> > Hequn
> >
> > On Fri, Nov 2, 2018 at 9:48 AM Shaoxuan Wang 
> wrote:
> >
> > > Hi Aljoscha，
> > > Glad that you like the proposal. We have completed the prototype of
> most
> > > new proposed functionalities. Once collect the feedback from community,
> > we
> > > will come up with a concrete FLIP/design doc.
> > >
> > > Regards,
> > > Shaoxuan
> > >
> > >
> > > On Thu, Nov 1, 2018 at 8:12 PM Aljoscha Krettek 
> > > wrote:
> > >
> > > > Hi Jincheng,
> > > >
> > > > these points sound very good! Are there any concrete proposals for
> > > > changes? For example a FLIP/design document?
> > > >
> > > > See here for FLIPs:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >
> > > > Best,
> > > > Aljoscha
> > > >
> > > > > On 1. Nov 2018, at 12:51, jincheng sun 
> > > wrote:
> > > > >
> > > > > *I am sorry for the formatting of the email content. I
> > reformat
> > > > > the **content** as follows---*
> > > > >
> > > > > *Hi ALL,*
> > > > >
> > > > > With the continuous efforts from the community, the Flink system
> has
> > > been
> > > > > continuously improved, which has attracted more and more users.
> Flink
> > > SQL
> > > > > is a canonical, widely used relational query language. However,
> there
> > > are
> > > > > still some scenarios where Flink SQL failed to meet user needs in
> > terms
> > > > of
> > > > > functionality and ease of use, such as:
> > > > >
> > > > > *1. In terms of functionality*
> > > > >Iteration, user-defined window, user-defined join, user-defined
> > > > > GroupReduce, etc. Users cannot express them with SQL;
> > > > >
> > > > > *2. In terms of ease of use*
> > > > >
> > > > >   - Map - e.g. “dataStream.map(mapFun)”. Although
> > “table.select(udf1(),
> > > > >   udf2(), udf3())” can be used to accomplish the same
> function.,
> > > > with a
> > > > >   map() function returning 100 columns, one has to define or call
> 100
> > > > UDFs
> > > > >   when using SQL, which is quite involved.
> > > > >   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. 

Why dont't have a csv formatter for kafka table source

[Jocean shi]

Hi all，
 I have  encountered a error When i want to register a table from kafka
using csv formatter.
 The error is "Could not find a suitable table factory for
'org.apache.flink.table.factories.DeserializationSchemaFactory"

Jocean

[jira] [Created] (FLINK-10760) Create a command line tool to migrate meta objects specified in SQL client configuration

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-10760:
---

 Summary: Create a command line tool to migrate meta objects 
specified in SQL client configuration
 Key: FLINK-10760
 URL: https://issues.apache.org/jira/browse/FLINK-10760
 Project: Flink
  Issue Type: Sub-task
  Components: SQL Client
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


With a persistent catalog for Flink meta objects (tables, views, functions, 
etc), it becomes unnecessary to specify such objects in SQL client 
configuration (YAML) file. However, it would be helpful to the users, who 
already have some meta objects specified in the YARM file, to have a command 
line tool that migrates objects specified in YAML files to the persistent 
catalog once for all. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10759) Adapt SQL-client configuration file to specify external catalogs and default catalog

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-10759:
---

 Summary: Adapt SQL-client configuration file to specify external 
catalogs and default catalog
 Key: FLINK-10759
 URL: https://issues.apache.org/jira/browse/FLINK-10759
 Project: Flink
  Issue Type: Sub-task
  Components: SQL Client
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


It doesn't seem that the configuration (YAML) file allows specifications of 
external catalogs currently. The request here is to add support for external 
catalog specifications in YAML file. User should also be able to specify one 
catalog is the default.

The catalog-related configurations then need to be processed and passed to 
{{TableEnvironment}} accordingly by calling relevant APIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10758) Refactor TableEnvironment so that all registration calls delegate to CatalogManager

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-10758:
---

 Summary: Refactor TableEnvironment so that all registration calls 
delegate to CatalogManager 
 Key: FLINK-10758
 URL: https://issues.apache.org/jira/browse/FLINK-10758
 Project: Flink
  Issue Type: Sub-task
  Components: Table API  SQL
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


There are many different APIs defined in {{TableEnvironment}} class that 
register tables/views/functions. Based on the design doc, those calls need to 
be delegated to {{CatalogManager}}. However, not all delegations are 
straightforward. For example. table registration could mean registering 
permanent tables, temp tables, or views. This JIRA takes care of the details. 

Please refer to the "TableEnvironment Class" section in the design doc 
(attached to the parent task) for more details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

Hi Hequn,
Thanks for your feedback! And also thanks for our offline discussion!
You are right, unification of batch and streaming is very important for
flink API.
We will provide more detailed design later, Please let me know if you have
further thoughts or feedback.

Thanks,
Jincheng

Hequn Cheng  于2018年11月2日周五 上午10:02写道：

> Hi Jincheng,
>
> Thanks a lot for your proposal. It is very encouraging!
>
> As we all know, SQL is a widely used language. It follows standards, is a
> descriptive language, and is easy to use. A powerful feature of SQL is that
> it supports optimization. Users only need to care about the logic of the
> program. The underlying optimizer will help users optimize the performance
> of the program. However, in terms of functionality and ease of use, in some
> scenarios sql will be limited, as described in Jincheng's proposal.
>
> Correspondingly, the DataStream/DataSet api can provide powerful
> functionalities. Users can write ProcessFunction/CoProcessFunction and get
> the timer. Compared with SQL, it provides more functionalities and
> flexibilities. However, it does not support optimization like SQL.
> Meanwhile, DataStream/DataSet api has not been unified which means, for the
> same logic, users need to write a job for each stream and batch.
>
> With TableApi, I think we can combine the advantages of both. Users can
> easily write relational operations and enjoy optimization. At the same
> time, it supports more functionality and ease of use. Looking forward to
> the detailed design/FLIP.
>
> Best,
> Hequn
>
> On Fri, Nov 2, 2018 at 9:48 AM Shaoxuan Wang  wrote:
>
> > Hi Aljoscha，
> > Glad that you like the proposal. We have completed the prototype of most
> > new proposed functionalities. Once collect the feedback from community,
> we
> > will come up with a concrete FLIP/design doc.
> >
> > Regards,
> > Shaoxuan
> >
> >
> > On Thu, Nov 1, 2018 at 8:12 PM Aljoscha Krettek 
> > wrote:
> >
> > > Hi Jincheng,
> > >
> > > these points sound very good! Are there any concrete proposals for
> > > changes? For example a FLIP/design document?
> > >
> > > See here for FLIPs:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > Best,
> > > Aljoscha
> > >
> > > > On 1. Nov 2018, at 12:51, jincheng sun 
> > wrote:
> > > >
> > > > *I am sorry for the formatting of the email content. I
> reformat
> > > > the **content** as follows---*
> > > >
> > > > *Hi ALL,*
> > > >
> > > > With the continuous efforts from the community, the Flink system has
> > been
> > > > continuously improved, which has attracted more and more users. Flink
> > SQL
> > > > is a canonical, widely used relational query language. However, there
> > are
> > > > still some scenarios where Flink SQL failed to meet user needs in
> terms
> > > of
> > > > functionality and ease of use, such as:
> > > >
> > > > *1. In terms of functionality*
> > > >Iteration, user-defined window, user-defined join, user-defined
> > > > GroupReduce, etc. Users cannot express them with SQL;
> > > >
> > > > *2. In terms of ease of use*
> > > >
> > > >   - Map - e.g. “dataStream.map(mapFun)”. Although
> “table.select(udf1(),
> > > >   udf2(), udf3())” can be used to accomplish the same function.,
> > > with a
> > > >   map() function returning 100 columns, one has to define or call 100
> > > UDFs
> > > >   when using SQL, which is quite involved.
> > > >   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it
> can
> > be
> > > >   implemented with “table.join(udtf).select()”. However, it is
> obvious
> > > that
> > > >   dataStream is easier to use than SQL.
> > > >
> > > > Due to the above two reasons, some users have to use the DataStream
> API
> > > or
> > > > the DataSet API. But when they do that, they lose the unification of
> > > batch
> > > > and streaming. They will also lose the sophisticated optimizations
> such
> > > as
> > > > codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> > > >
> > > > We believe that enhancing the functionality and productivity is vital
> > for
> > > > the successful adoption of Table API. To this end,  Table API still
> > > > requires more efforts from every contributor in the community. We see
> > > great
> > > > opportunity in improving our user’s experience from this work. Any
> > > feedback
> > > > is welcome.
> > > >
> > > > Regards,
> > > >
> > > > Jincheng
> > > >
> > > > jincheng sun  于2018年11月1日周四 下午5:07写道：
> > > >
> > > >> Hi all,
> > > >>
> > > >> With the continuous efforts from the community, the Flink system has
> > > been
> > > >> continuously improved, which has attracted more and more users.
> Flink
> > > SQL
> > > >> is a canonical, widely used relational query language. However,
> there
> > > are
> > > >> still some scenarios where Flink SQL failed to meet user needs in
> > terms
> > > of
> > > >> functionality and ease of use, such as:
> > > >>
> > > >>
> > > >>   -
> > > >>
> > > >>   In terms 

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

Hi, Jiangjie,
Thanks a lot for your feedback. And also thanks for our offline discussion!
Yes, your right! The Row-based APIs which you mentioned are very friendly
to flink user!
In order to follow the concept of the traditional database, perhaps we
named the corresponding function RowValued/TabeValued function will be more
appropriate, then from the perspective of return value in TableAPI we have
three type functions:

   - ColumnValuedFunction - ScalarFunction & AggregateFunction, and the
   result is a column.
   - RowValuedFunction - MapFunction which I'll proposal is
   RowValuedFunciton, and result is a single row.
   - TableValuedFunction - FlatMapFunction which I'll proposal is
   TableValuedFunciton, and result is a table.

The detail will be described in following FLIP/design doc.
About the input type I think we can support both column parameters and row
parameters. but I think the meaning you want to express should be
consistent with me, we are on the same page, right?

And thanks you like the proposal, I hope that we can work together to
advance the work!

Best,
Jincheng

Becket Qin  于2018年11月2日周五 上午1:25写道：

> Thanks for the proposal, Jincheng.
>
> This makes a lot of sense. As a programming interface, Table API is
> especially attractive because it supports both batch and stream. However,
> the relational-only API often forces users to shoehorn their logic into a
> bunch of user defined functions. Introducing some more flexible API (e.g.
> row-based APIs) to process records would really help here.
>
> Besides the processing API, another useful improvement would be allowing
> batch tables and stream tables to run in the same job, which is actually a
> quite common scenario.
>
> As you said, there are a lot of work could be done here. I am looking
> forward to the upcoming FLIPs.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Nov 2, 2018 at 12:10 AM jincheng sun 
> wrote:
>
> > Hi, Timo,
> > I am very grateful for your feedback, and I am very excited when I hear
> > that you also consider adding a process function to the TableAPI.
> >
> > I agree that add support for the Process Function on the Table API, which
> > is actually part of my proposal Enhancing the functionality of Table API.
> > In fact, supporting the ProcessFunction means supporting the user-defined
> > Operator. As you said, A ProcessFunction can implement any logic,
> including
> > the user-defined window, which leaves the user with enough freedom and
> > control. At the same time, Co-PrecessFunction needs to be supported, so
> we
> > can implement the logic of User-Defined JOIN through Co-PrecessFunciton.
> Of
> > course, Co-PrecessFunciton also needs to introduce the concept of
> Connect,
> > and will introduce a new ConnectedTable type on TableAPI.  And I also
> think
> > TableAPI also for more event-driven applications.
> >
> > About processFunction In addition to the timer function, it should be
> > completely equivalent to flatmapFunction, so maybe we can support map and
> > flatmap in Table, support processFunction in GroupedTable, because for
> the
> > reason of State, the Timer of ProcessFunction can only Apply to
> > KeyedStream.
> >
> > You are right, ANSI-SQL is difficult to express complex operator logic
> such
> > as ProcessFunction, so once we decide to make these enhancements on the
> > TableAPI, it means that the Flink SQL only includes ANSI-SQL operations,
> > and the TableAPI' operations is SQL super set. This means that the Flink
> > High-level API includes the A Query language SQL and A powerfu program
> > language Table API. In this way, SQL using for those simple ETL user
> > groups, the TableAPI is for a user group that needs to be customized for
> > complex logic, and these users can enjoy The benefit of the query
> > optimizer. Maybe we need more refinement and hard work to support these
> > functions, but maybe this is a good direction of effort.
> >
> > Thanks,
> > Jincheng
> >
> > Timo Walther  于2018年11月1日周四 下午10:08写道：
> >
> > > Hi Jincheng,
> > >
> > > I was also thinking about introducing a process function for the Table
> > > API several times. This would allow to define more complex logic
> (custom
> > > windows, timers, etc.) embedded into a relational API with schema
> > > awareness and optimization around the black box. Of course this would
> > > mean that we diverge with Table API from SQL API, however, it would
> open
> > > the Table API also for more event-driven applications.
> > >
> > > Maybe it would be possible to define timers and firing logic using
> Table
> > > API expressions and UDFs. Within planning this would be treated as a
> > > special Calc node.
> > >
> > > Just some ideas that might be interesting for new use cases.
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > > Am 01.11.18 um 13:12 schrieb Aljoscha Krettek:
> > > > Hi Jincheng,
> > > >
> > > > these points sound very good! Are there any concrete proposals for
> > > changes? For example a FLIP/design document?
> > > >

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

[Shaoxuan Wang]

+1 to keep the web UI stack to Angular. This will make the stack upgrading
much easier and won't break the on-going efforts already made towards the
improvements of Flink Web UI.

Till, Robert, and Fabian,
Please take a look at the upgrading plan from Angular 1.x to 7.0 proposed
by Yadong. Any feedback is appreciated.

Regards,
Shaoxuan


On Thu, Nov 1, 2018 at 5:41 PM Yadong Xie  wrote:

> For your interest, here are some snapshots of Flink web UI (with angular
> 7.0) we recently reconstructed.
> [image: snapshot.001.jpeg]
>
> https://drive.google.com/file/d/1p7rcmgHi3ZJpjt7h1H97VIrIhNkiBs1j/view?usp=sharing
>
> We choose Angular 7.0 due to the following reasons:
>
>1. It is easy to migrate the current codebase to Angular 7.0 following
>the official guide.
>2. It is easy for current Web UI developers who are experienced with
>Angular 1.x to get familiar with 7.0 reading the official doc.
>3. Migrating other than rewriting is helpful to ensure its functional
>integrity.
>4. There are over 1 million Angular developers now (2.0 and above,
>still increasing rapidly), which means there are potentially more
>developers willing to join the community.
>5. No bower or complex node / webpack configuration needed (only
>angular cli).
>
> We split Angular 1.x and 7.0 versions into two folders based on the
> following consideration:
>
>1. 1.x version could be the default page because there is no
>breakchange for both users and developers.
>2. Make the upgrade progressive. Users can switch between two versions
>by clicking the button on the top right corner.
>3. Reduce maintenance costs and keep the performance of web without
>bootstrap both 1.x and 7.0 versions in the same page.
>4. Won't interrupt the current development process. The code can still
>be submitted to 1.x version folder, we could help to sync all features and
>bug fix to 7.0 version if needed.
>5. When the 7.0 version is ready and stable, it is easy to change
>default page to it, and remove the old version.
>
> [image: test.jpg]
>
>
> https://drive.google.com/file/d/1v7U5upP0fSepVl-E_46k-Qfj9uVfrGZf/view?usp=sharing
>
> Here is the brief list of what we have done:
>
>1. Create a new folder named next-web-dashboard in flink-runtime-web.
>2. Migrate all Angular 1.x controllers, services, config etc one by
>one into 7.0 version with ngMigration.
>3. Modify frontend build scripts in order to package two versions
>together.
>4. All works listed above are limited in the flink-runtime-web folder.
>One folder was added and no file or folder was deleted.
>
> The new version of the Web UI is running stable for several months in
> Alibaba. We are happy to contribute it to the Flink community if everyone
> like the work.
>
> Best
> Yadong
>
> On Wed, Oct 31, 2018 at 5:48 PM Till Rohrmann 
> wrote:
>
>> I agree with Chesnay that the web UI is already fairly isolated in Flink's
>> code base (see flink-runtime-web module). Moreover, imo it is very
>> important that Flink comes with a nice web UI which works out of the box
>> because it lowers the entrance hurdle massively. What one could argue is
>> to
>> make it easier to plug in your own web UI if you want to develop it. But I
>> would see this as an orthogonal issue.
>>
>> I would actually argue against Robert's proposal to make a grand rewrite
>> if
>> this is by any means possible. Experience shows that these kind of efforts
>> are usually set up to fail or take super long. If possible, then I would
>> rather prefer to convert one component at a time to React. But I'm not an
>> expert here and cannot tell whether this is possible or not.
>>
>> What we would need is active help from the community to review these
>> changes. I think that is the crucial piece to make such an effort work or
>> fail.
>>
>> Cheers,
>> Till
>>
>> On Wed, Oct 31, 2018 at 10:14 AM Chesnay Schepler 
>> wrote:
>>
>> > The WebUI goes entirely through the REST API.
>> >
>> > I don't see how moving it out would make it easier to
>> develop/contribute.
>> >
>> > The UI is already encapsulated in it's own module (flink-runtime-web)
>> > and changes can be made to it independently.
>> > Historically the issue was simply that contributions to the UI were not
>> > reviewed in a timely manner /  not at all, which mind you
>> > applies to virtually all components. This also won't change by moving it
>> > out of Flink.
>> >
>> > Having it in Flink also guarantees that there is an easily-discoverable
>> > version of the UI that is compatible with your Flink distribution.
>> >
>> > On 31.10.2018 09:53, dyana.rose wrote:
>> > > Re: who's using the web ui
>> > >
>> > > Though many mature solutions, with a fair amount of time/resources
>> > available are likely running their own front ends, for teams like mine
>> > which are smaller and aren't focused solely on working with Flink,
>> having
>> > the web ui available removes a large barrier 

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Hequn Cheng]

Hi Jincheng,

Thanks a lot for your proposal. It is very encouraging!

As we all know, SQL is a widely used language. It follows standards, is a
descriptive language, and is easy to use. A powerful feature of SQL is that
it supports optimization. Users only need to care about the logic of the
program. The underlying optimizer will help users optimize the performance
of the program. However, in terms of functionality and ease of use, in some
scenarios sql will be limited, as described in Jincheng's proposal.

Correspondingly, the DataStream/DataSet api can provide powerful
functionalities. Users can write ProcessFunction/CoProcessFunction and get
the timer. Compared with SQL, it provides more functionalities and
flexibilities. However, it does not support optimization like SQL.
Meanwhile, DataStream/DataSet api has not been unified which means, for the
same logic, users need to write a job for each stream and batch.

With TableApi, I think we can combine the advantages of both. Users can
easily write relational operations and enjoy optimization. At the same
time, it supports more functionality and ease of use. Looking forward to
the detailed design/FLIP.

Best,
Hequn

On Fri, Nov 2, 2018 at 9:48 AM Shaoxuan Wang  wrote:

> Hi Aljoscha，
> Glad that you like the proposal. We have completed the prototype of most
> new proposed functionalities. Once collect the feedback from community, we
> will come up with a concrete FLIP/design doc.
>
> Regards,
> Shaoxuan
>
>
> On Thu, Nov 1, 2018 at 8:12 PM Aljoscha Krettek 
> wrote:
>
> > Hi Jincheng,
> >
> > these points sound very good! Are there any concrete proposals for
> > changes? For example a FLIP/design document?
> >
> > See here for FLIPs:
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > Best,
> > Aljoscha
> >
> > > On 1. Nov 2018, at 12:51, jincheng sun 
> wrote:
> > >
> > > *I am sorry for the formatting of the email content. I reformat
> > > the **content** as follows---*
> > >
> > > *Hi ALL,*
> > >
> > > With the continuous efforts from the community, the Flink system has
> been
> > > continuously improved, which has attracted more and more users. Flink
> SQL
> > > is a canonical, widely used relational query language. However, there
> are
> > > still some scenarios where Flink SQL failed to meet user needs in terms
> > of
> > > functionality and ease of use, such as:
> > >
> > > *1. In terms of functionality*
> > >Iteration, user-defined window, user-defined join, user-defined
> > > GroupReduce, etc. Users cannot express them with SQL;
> > >
> > > *2. In terms of ease of use*
> > >
> > >   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> > >   udf2(), udf3())” can be used to accomplish the same function.,
> > with a
> > >   map() function returning 100 columns, one has to define or call 100
> > UDFs
> > >   when using SQL, which is quite involved.
> > >   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
> be
> > >   implemented with “table.join(udtf).select()”. However, it is obvious
> > that
> > >   dataStream is easier to use than SQL.
> > >
> > > Due to the above two reasons, some users have to use the DataStream API
> > or
> > > the DataSet API. But when they do that, they lose the unification of
> > batch
> > > and streaming. They will also lose the sophisticated optimizations such
> > as
> > > codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> > >
> > > We believe that enhancing the functionality and productivity is vital
> for
> > > the successful adoption of Table API. To this end,  Table API still
> > > requires more efforts from every contributor in the community. We see
> > great
> > > opportunity in improving our user’s experience from this work. Any
> > feedback
> > > is welcome.
> > >
> > > Regards,
> > >
> > > Jincheng
> > >
> > > jincheng sun  于2018年11月1日周四 下午5:07写道：
> > >
> > >> Hi all,
> > >>
> > >> With the continuous efforts from the community, the Flink system has
> > been
> > >> continuously improved, which has attracted more and more users. Flink
> > SQL
> > >> is a canonical, widely used relational query language. However, there
> > are
> > >> still some scenarios where Flink SQL failed to meet user needs in
> terms
> > of
> > >> functionality and ease of use, such as:
> > >>
> > >>
> > >>   -
> > >>
> > >>   In terms of functionality
> > >>
> > >> Iteration, user-defined window, user-defined join, user-defined
> > >> GroupReduce, etc. Users cannot express them with SQL;
> > >>
> > >>   -
> > >>
> > >>   In terms of ease of use
> > >>   -
> > >>
> > >>  Map - e.g. “dataStream.map(mapFun)”. Although
> “table.select(udf1(),
> > >>  udf2(), udf3())” can be used to accomplish the same
> function.,
> > with a
> > >>  map() function returning 100 columns, one has to define or call
> > 100 UDFs
> > >>  when using SQL, which is quite involved.
> > >>  -
> > >>
> > >>  FlatMap -  e.g. 

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Shaoxuan Wang]

Hi Aljoscha，
Glad that you like the proposal. We have completed the prototype of most
new proposed functionalities. Once collect the feedback from community, we
will come up with a concrete FLIP/design doc.

Regards,
Shaoxuan


On Thu, Nov 1, 2018 at 8:12 PM Aljoscha Krettek  wrote:

> Hi Jincheng,
>
> these points sound very good! Are there any concrete proposals for
> changes? For example a FLIP/design document?
>
> See here for FLIPs:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> Best,
> Aljoscha
>
> > On 1. Nov 2018, at 12:51, jincheng sun  wrote:
> >
> > *I am sorry for the formatting of the email content. I reformat
> > the **content** as follows---*
> >
> > *Hi ALL,*
> >
> > With the continuous efforts from the community, the Flink system has been
> > continuously improved, which has attracted more and more users. Flink SQL
> > is a canonical, widely used relational query language. However, there are
> > still some scenarios where Flink SQL failed to meet user needs in terms
> of
> > functionality and ease of use, such as:
> >
> > *1. In terms of functionality*
> >Iteration, user-defined window, user-defined join, user-defined
> > GroupReduce, etc. Users cannot express them with SQL;
> >
> > *2. In terms of ease of use*
> >
> >   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> >   udf2(), udf3())” can be used to accomplish the same function.,
> with a
> >   map() function returning 100 columns, one has to define or call 100
> UDFs
> >   when using SQL, which is quite involved.
> >   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
> >   implemented with “table.join(udtf).select()”. However, it is obvious
> that
> >   dataStream is easier to use than SQL.
> >
> > Due to the above two reasons, some users have to use the DataStream API
> or
> > the DataSet API. But when they do that, they lose the unification of
> batch
> > and streaming. They will also lose the sophisticated optimizations such
> as
> > codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> >
> > We believe that enhancing the functionality and productivity is vital for
> > the successful adoption of Table API. To this end,  Table API still
> > requires more efforts from every contributor in the community. We see
> great
> > opportunity in improving our user’s experience from this work. Any
> feedback
> > is welcome.
> >
> > Regards,
> >
> > Jincheng
> >
> > jincheng sun  于2018年11月1日周四 下午5:07写道：
> >
> >> Hi all,
> >>
> >> With the continuous efforts from the community, the Flink system has
> been
> >> continuously improved, which has attracted more and more users. Flink
> SQL
> >> is a canonical, widely used relational query language. However, there
> are
> >> still some scenarios where Flink SQL failed to meet user needs in terms
> of
> >> functionality and ease of use, such as:
> >>
> >>
> >>   -
> >>
> >>   In terms of functionality
> >>
> >> Iteration, user-defined window, user-defined join, user-defined
> >> GroupReduce, etc. Users cannot express them with SQL;
> >>
> >>   -
> >>
> >>   In terms of ease of use
> >>   -
> >>
> >>  Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> >>  udf2(), udf3())” can be used to accomplish the same function.,
> with a
> >>  map() function returning 100 columns, one has to define or call
> 100 UDFs
> >>  when using SQL, which is quite involved.
> >>  -
> >>
> >>  FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
> >>  be implemented with “table.join(udtf).select()”. However, it is
> obvious
> >>  that datastream is easier to use than SQL.
> >>
> >>
> >> Due to the above two reasons, some users have to use the DataStream API
> or
> >> the DataSet API. But when they do that, they lose the unification of
> batch
> >> and streaming. They will also lose the sophisticated optimizations such
> as
> >> codegen, aggregate join transpose  and multi-stage agg from Flink SQL.
> >>
> >> We believe that enhancing the functionality and productivity is vital
> for
> >> the successful adoption of Table API. To this end,  Table API still
> >> requires more efforts from every contributor in the community. We see
> great
> >> opportunity in improving our user’s experience from this work. Any
> feedback
> >> is welcome.
> >>
> >> Regards,
> >>
> >> Jincheng
> >>
> >>
>
>

[jira] [Created] (FLINK-10757) TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.

[Bowen Li (JIRA)]

Bowen Li created FLINK-10757:


 Summary: TaskManagerProcessFailureStreamingRecoveryITCase failed 
to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.
 Key: FLINK-10757
 URL: https://issues.apache.org/jira/browse/FLINK-10757
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.6.2, 1.7.0
Reporter: Bowen Li
 Fix For: 1.7.0


{code:java}
Failed tests: 
  ...
 
TaskManagerProcessFailureStreamingRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure:223
 Failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.
Tests in error: 
{code}


https://travis-ci.org/apache/flink/jobs/449439623



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10756) TaskManagerProcessFailureBatchRecoveryITCase did not finish on time

[Bowen Li (JIRA)]

Bowen Li created FLINK-10756:


 Summary: TaskManagerProcessFailureBatchRecoveryITCase did not 
finish on time
 Key: FLINK-10756
 URL: https://issues.apache.org/jira/browse/FLINK-10756
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.7.0
Reporter: Bowen Li
 Fix For: 1.7.0


{code:java}
Failed tests: 
  
TaskManagerProcessFailureBatchRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure:207
 The program did not finish in time
{code}

https://travis-ci.org/apache/flink/jobs/449439623



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Demo of the K8S Flink Operator at the K8s Big Data SIG

[Anand Swaminathan]

Thank you all for providing us the opportunity to present our work.

Here is the presentation

that we used.

Also, you can find more information about FlinkK8sOperator here


Please reach out us for more information. We are happy to collaborate with
you.

Thanks,
Anand

On Wed, Oct 31, 2018 at 1:54 PM Erik Erlandson  wrote:

> This week at the K8S Big Data SIG we had a demo of a Flink K8s operator
> from Anand Swaminathan and Ketan Umare of Lyft. The talk was recorded, and
> the link is here:
>
> https://zoom.us/recording/share/T-MWuMwfY-6S3Y4lfu9lrLYOQwJGQmxdsBlFZlNws7CwIumekTziMw?startTime=1541005448000
>
> Thanks Anand and Ketan for the great talk and demo!
> Cheers,
> Erik Erlandson
>

Re: Flink operators for Kubernetes

[Anand Swaminathan]

Hello,

I have documented our work of running Flink applications on Kubernetes
using FlinkK8sOperator - here


We have finished the implementation of the FlinkK8sOperator, and are
currently testing it with few applications. Please let us know your
feedback.

Thanks,
Anand
(Flyte @Lyft)


On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann  wrote:

> Great to hear that you intend to open source your K8s operators. I would be
> keen to see what and how you do things with the operator. If there are
> things to change on the Flink side in order to improve the integration,
> then let's discuss them.
>
> Cheers,
> Till
>
> On Wed, Oct 3, 2018 at 2:52 AM Jin Sun  wrote:
>
> > Sounds interesting.
> >
> >
> >
> > Currently Flink can run in standalone mode in a kubernets cluster,
> details
> > see here.   There is also a JIRA Flink-9953: Active kubernetes
> integration
> > is target for run Flink on Kubernetes natively.
> >
> >
> >
> > Jin
> >
> >
> >
> > On 10/2/18, 1:30 PM, "Anand Swaminathan"  >
> > wrote:
> >
> >
> >
> > Hello All,
> >
> >
> >
> > This is Anand from Lyft. Just wanted to send out a note that we at
> > Lyft are
> >
> > working on building Operators (https://coreos.com/operators/) to
> > support
> >
> > deploying, managing Flink Applications in Kubernetes. The operator is
> >
> > responsible for creating Flink Clusters, start jobs and seamlessly
> >
> > transition a job from one cluster to another during deployment.
> >
> >
> >
> > We hope to open source it at some point once deployed and tested
> within
> >
> > Lyft. Let us know if any of you are looking for something like this.
> We
> >
> > will be happy to collaborate.
> >
> >
> >
> > Thanks,
> >
> > Anand
> >
> > (Flyte @Lyft)
> >
> >
> >
> >
>

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

[Bowen Li]

After taking a look at how other discussion threads work, I think it's
actually fine just keep our discussion here. It's up to you, Xuefu.

The google doc LGTM. I left some minor comments.

On Thu, Nov 1, 2018 at 10:17 AM Bowen Li  wrote:

> Hi all,
>
> As Xuefu has published the design doc on google, I agree with Shuyi's
> suggestion that we probably should start a new email thread like "[DISCUSS]
> ... Hive integration design ..." on only dev mailing list for community
> devs to review. The current thread sends to both dev and user list.
>
> This email thread is more like validating the general idea and direction
> with the community, and it's been pretty long and crowded so far. Since
> everyone is pro for the idea, we can move forward with another thread to
> discuss and finalize the design.
>
> Thanks,
> Bowen
>
> On Wed, Oct 31, 2018 at 12:16 PM Zhang, Xuefu 
> wrote:
>
>> Hi Shuiyi,
>>
>> Good idea. Actually the PDF was converted from a google doc. Here is its
>> link:
>>
>> https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
>> Once we reach an agreement, I can convert it to a FLIP.
>>
>> Thanks,
>> Xuefu
>>
>>
>>
>> --
>> Sender:Shuyi Chen 
>> Sent at:2018 Nov 1 (Thu) 02:47
>> Recipient:Xuefu 
>> Cc:vino yang ; Fabian Hueske ;
>> dev ; user 
>> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>>
>> Hi Xuefu,
>>
>> Thanks a lot for driving this big effort. I would suggest convert your
>> proposal and design doc into a google doc, and share it on the dev mailing
>> list for the community to review and comment with title like "[DISCUSS] ...
>> Hive integration design ..." . Once approved,  we can document it as a FLIP
>> (Flink Improvement Proposals), and use JIRAs to track the implementations.
>> What do you think?
>>
>> Shuyi
>>
>> On Tue, Oct 30, 2018 at 11:32 AM Zhang, Xuefu 
>> wrote:
>> Hi all,
>>
>> I have also shared a design doc on Hive metastore integration that is
>> attached here and also to FLINK-10556[1]. Please kindly review and share
>> your feedback.
>>
>>
>> Thanks,
>> Xuefu
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-10556
>> --
>> Sender:Xuefu 
>> Sent at:2018 Oct 25 (Thu) 01:08
>> Recipient:Xuefu ; Shuyi Chen > >
>> Cc:yanghua1127 ; Fabian Hueske ;
>> dev ; user 
>> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>>
>> Hi all,
>>
>> To wrap up the discussion, I have attached a PDF describing the proposal,
>> which is also attached to FLINK-10556 [1]. Please feel free to watch that
>> JIRA to track the progress.
>>
>> Please also let me know if you have additional comments or questions.
>>
>> Thanks,
>> Xuefu
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-10556
>>
>>
>> --
>> Sender:Xuefu 
>> Sent at:2018 Oct 16 (Tue) 03:40
>> Recipient:Shuyi Chen 
>> Cc:yanghua1127 ; Fabian Hueske ;
>> dev ; user 
>> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>>
>> Hi Shuyi,
>>
>> Thank you for your input. Yes, I agreed with a phased approach and like
>> to move forward fast. :) We did some work internally on DDL utilizing babel
>> parser in Calcite. While babel makes Calcite's grammar extensible, at
>> first impression it still seems too cumbersome for a project when too
>> much extensions are made. It's even challenging to find where the extension
>> is needed! It would be certainly better if Calcite can magically support
>> Hive QL by just turning on a flag, such as that for MYSQL_5. I can also
>> see that this could mean a lot of work on Calcite. Nevertheless, I will
>> bring up the discussion over there and to see what their community thinks.
>>
>> Would mind to share more info about the proposal on DDL that you
>> mentioned? We can certainly collaborate on this.
>>
>> Thanks,
>> Xuefu
>>
>> --
>> Sender:Shuyi Chen 
>> Sent at:2018 Oct 14 (Sun) 08:30
>> Recipient:Xuefu 
>> Cc:yanghua1127 ; Fabian Hueske ;
>> dev ; user 
>> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>>
>> Welcome to the community and thanks for the great proposal, Xuefu! I
>> think the proposal can be divided into 2 stages: making Flink to support
>> Hive features, and make Hive to work with Flink. I agreed with Timo that on
>> starting with a smaller scope, so we can make progress faster. As for [6],
>> a proposal for DDL is already in progress, and will come after the unified
>> SQL connector API is done. For supporting Hive syntax, we might need to
>> work with the Calcite community, and a recent effort called babel (
>> https://issues.apache.org/jira/browse/CALCITE-2280) in Calcite might
>> help here.
>>
>> Thanks
>> Shuyi
>>
>> On Wed, Oct 10, 2018 at 8:02 PM Zhang, Xuefu 
>> wrote:
>> Hi Fabian/Vno,
>>
>> Thank you very 

[jira] [Created] (FLINK-10755) Port external catalogs in Table API extension points to flink-table-common

[Bowen Li (JIRA)]

Bowen Li created FLINK-10755:


 Summary: Port external catalogs in Table API extension points to 
flink-table-common
 Key: FLINK-10755
 URL: https://issues.apache.org/jira/browse/FLINK-10755
 Project: Flink
  Issue Type: Sub-task
  Components: Table API  SQL
Reporter: Bowen Li
Assignee: xueyu


After FLINK-10687 and FLINK-10688 have been resolved, we should also port the 
remaining extension points of the Table API to flink-table-common. This 
includes interfaces for UDFs and the external catalog interface.

This ticket is for porting UDFs. This jira does NOT depend on FLINK-16088 so it 
can be started at anytime.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Becket Qin]

Thanks for the FLIP, Aljoscha.

The proposal makes sense to me. Separating the split discovery and
consumption is very useful as it enables Flink to better manage the sources.

Looking at the interface, I have a few questions:
1. *SplitEnumerator*.*discoverNewSplits()* seems assuming that the number
of splits can only increase, In your example, the source was Kafka, so the
assumption was true. But I am wondering are there case that the number of
splits can decrease?
2. I agree with Piotr that we need to be careful about potentially blocking
implementations. However, it is not clear to me how does the completable
future work if the underlying reader does not have its own thread (e.g. a
Kafka consumer). In that case, the future will never be completed unless
the caller thread touches the reader again. I am wondering if the following
interfaces for the reader makes sense:
boolean isDone(); // Whether the source has more records.
T poll(); // non-blocking read. We can add a timeout if needed.
T take(); // blocking read;
This seems more intuitive to people who are familiar with existing
convention of poll() and take(). And with the non-blocking poll() we could
have an nio Selector-like API when there are multiple splits.

BTW, it would be really helpful if there is some Java doc describing the
behavior of the the interfaces in the FLIP.

Thanks again for the great proposal.

Jiangjie (Becket) Qin

On Thu, Nov 1, 2018 at 6:08 PM Piotr Nowojski 
wrote:

> Hi,
>
> Thanks Aljoscha for starting this, it’s blocking quite a lot of other
> possible improvements. I have one proposal. Instead of having a method:
>
> boolean advance() throws IOException;
>
> I would replace it with
>
> /*
>  * Return a future, which when completed means that source has more data
> and getNext() will not block.
>  * If you wish to use benefits of non blocking connectors, please
> implement this method appropriately.
>  */
> default CompletableFuture isBlocked() {
> return CompletableFuture.completedFuture(null);
> }
>
> And rename `getCurrent()` to `getNext()`.
>
> Couple of arguments:
> 1. I don’t understand the division of work between `advance()` and
> `getCurrent()`. What should be done in which, especially for connectors
> that handle records in batches (like Kafka) and when should you call
> `advance` and when `getCurrent()`.
> 2. Replacing `boolean` with `CompletableFuture` will allow us in the
> future to have asynchronous/non blocking connectors and more efficiently
> handle large number of blocked threads, without busy waiting. While at the
> same time it doesn’t add much complexity, since naive connector
> implementations can be always blocking.
> 3. This also would allow us to use a fixed size thread pool of task
> executors, instead of one thread per task.
>
> Piotrek
>
> > On 31 Oct 2018, at 17:22, Aljoscha Krettek  wrote:
> >
> > Hi All,
> >
> > In order to finally get the ball rolling on the new source interface
> that we have discussed for so long I finally created a FLIP:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >
> > I cc'ed Thomas and Jamie because of the ongoing work/discussion about
> adding per-partition watermark support to the Kinesis source and because
> this would enable generic implementation of event-time alignment for all
> sources. Maybe we need another FLIP for the event-time alignment part,
> especially the part about information sharing between operations (I'm not
> calling it state sharing because state has a special meaning in Flink).
> >
> > Please discuss away!
> >
> > Aljoscha
> >
> >
>
>

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Becket Qin]

Thanks for the proposal, Jincheng.

This makes a lot of sense. As a programming interface, Table API is
especially attractive because it supports both batch and stream. However,
the relational-only API often forces users to shoehorn their logic into a
bunch of user defined functions. Introducing some more flexible API (e.g.
row-based APIs) to process records would really help here.

Besides the processing API, another useful improvement would be allowing
batch tables and stream tables to run in the same job, which is actually a
quite common scenario.

As you said, there are a lot of work could be done here. I am looking
forward to the upcoming FLIPs.

Thanks,

Jiangjie (Becket) Qin

On Fri, Nov 2, 2018 at 12:10 AM jincheng sun 
wrote:

> Hi, Timo,
> I am very grateful for your feedback, and I am very excited when I hear
> that you also consider adding a process function to the TableAPI.
>
> I agree that add support for the Process Function on the Table API, which
> is actually part of my proposal Enhancing the functionality of Table API.
> In fact, supporting the ProcessFunction means supporting the user-defined
> Operator. As you said, A ProcessFunction can implement any logic, including
> the user-defined window, which leaves the user with enough freedom and
> control. At the same time, Co-PrecessFunction needs to be supported, so we
> can implement the logic of User-Defined JOIN through Co-PrecessFunciton. Of
> course, Co-PrecessFunciton also needs to introduce the concept of Connect,
> and will introduce a new ConnectedTable type on TableAPI.  And I also think
> TableAPI also for more event-driven applications.
>
> About processFunction In addition to the timer function, it should be
> completely equivalent to flatmapFunction, so maybe we can support map and
> flatmap in Table, support processFunction in GroupedTable, because for the
> reason of State, the Timer of ProcessFunction can only Apply to
> KeyedStream.
>
> You are right, ANSI-SQL is difficult to express complex operator logic such
> as ProcessFunction, so once we decide to make these enhancements on the
> TableAPI, it means that the Flink SQL only includes ANSI-SQL operations,
> and the TableAPI' operations is SQL super set. This means that the Flink
> High-level API includes the A Query language SQL and A powerfu program
> language Table API. In this way, SQL using for those simple ETL user
> groups, the TableAPI is for a user group that needs to be customized for
> complex logic, and these users can enjoy The benefit of the query
> optimizer. Maybe we need more refinement and hard work to support these
> functions, but maybe this is a good direction of effort.
>
> Thanks,
> Jincheng
>
> Timo Walther  于2018年11月1日周四 下午10:08写道：
>
> > Hi Jincheng,
> >
> > I was also thinking about introducing a process function for the Table
> > API several times. This would allow to define more complex logic (custom
> > windows, timers, etc.) embedded into a relational API with schema
> > awareness and optimization around the black box. Of course this would
> > mean that we diverge with Table API from SQL API, however, it would open
> > the Table API also for more event-driven applications.
> >
> > Maybe it would be possible to define timers and firing logic using Table
> > API expressions and UDFs. Within planning this would be treated as a
> > special Calc node.
> >
> > Just some ideas that might be interesting for new use cases.
> >
> > Regards,
> > Timo
> >
> >
> > Am 01.11.18 um 13:12 schrieb Aljoscha Krettek:
> > > Hi Jincheng,
> > >
> > > these points sound very good! Are there any concrete proposals for
> > changes? For example a FLIP/design document?
> > >
> > > See here for FLIPs:
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > Best,
> > > Aljoscha
> > >
> > >> On 1. Nov 2018, at 12:51, jincheng sun 
> > wrote:
> > >>
> > >> *I am sorry for the formatting of the email content. I
> reformat
> > >> the **content** as follows---*
> > >>
> > >> *Hi ALL,*
> > >>
> > >> With the continuous efforts from the community, the Flink system has
> > been
> > >> continuously improved, which has attracted more and more users. Flink
> > SQL
> > >> is a canonical, widely used relational query language. However, there
> > are
> > >> still some scenarios where Flink SQL failed to meet user needs in
> terms
> > of
> > >> functionality and ease of use, such as:
> > >>
> > >> *1. In terms of functionality*
> > >> Iteration, user-defined window, user-defined join, user-defined
> > >> GroupReduce, etc. Users cannot express them with SQL;
> > >>
> > >> *2. In terms of ease of use*
> > >>
> > >>- Map - e.g. “dataStream.map(mapFun)”. Although
> “table.select(udf1(),
> > >>udf2(), udf3())” can be used to accomplish the same function.,
> > with a
> > >>map() function returning 100 columns, one has to define or call 100
> > UDFs
> > >>when using SQL, which is quite involved.
> > >>- 

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

[Bowen Li]

Hi all,

As Xuefu has published the design doc on google, I agree with Shuyi's
suggestion that we probably should start a new email thread like "[DISCUSS]
... Hive integration design ..." on only dev mailing list for community
devs to review. The current thread sends to both dev and user list.

This email thread is more like validating the general idea and direction
with the community, and it's been pretty long and crowded so far. Since
everyone is pro for the idea, we can move forward with another thread to
discuss and finalize the design.

Thanks,
Bowen

On Wed, Oct 31, 2018 at 12:16 PM Zhang, Xuefu 
wrote:

> Hi Shuiyi,
>
> Good idea. Actually the PDF was converted from a google doc. Here is its
> link:
>
> https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
> Once we reach an agreement, I can convert it to a FLIP.
>
> Thanks,
> Xuefu
>
>
>
> --
> Sender:Shuyi Chen 
> Sent at:2018 Nov 1 (Thu) 02:47
> Recipient:Xuefu 
> Cc:vino yang ; Fabian Hueske ;
> dev ; user 
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi Xuefu,
>
> Thanks a lot for driving this big effort. I would suggest convert your
> proposal and design doc into a google doc, and share it on the dev mailing
> list for the community to review and comment with title like "[DISCUSS] ...
> Hive integration design ..." . Once approved,  we can document it as a FLIP
> (Flink Improvement Proposals), and use JIRAs to track the implementations.
> What do you think?
>
> Shuyi
>
> On Tue, Oct 30, 2018 at 11:32 AM Zhang, Xuefu 
> wrote:
> Hi all,
>
> I have also shared a design doc on Hive metastore integration that is
> attached here and also to FLINK-10556[1]. Please kindly review and share
> your feedback.
>
>
> Thanks,
> Xuefu
>
> [1] https://issues.apache.org/jira/browse/FLINK-10556
> --
> Sender:Xuefu 
> Sent at:2018 Oct 25 (Thu) 01:08
> Recipient:Xuefu ; Shuyi Chen 
> Cc:yanghua1127 ; Fabian Hueske ;
> dev ; user 
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi all,
>
> To wrap up the discussion, I have attached a PDF describing the proposal,
> which is also attached to FLINK-10556 [1]. Please feel free to watch that
> JIRA to track the progress.
>
> Please also let me know if you have additional comments or questions.
>
> Thanks,
> Xuefu
>
> [1] https://issues.apache.org/jira/browse/FLINK-10556
>
>
> --
> Sender:Xuefu 
> Sent at:2018 Oct 16 (Tue) 03:40
> Recipient:Shuyi Chen 
> Cc:yanghua1127 ; Fabian Hueske ;
> dev ; user 
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Hi Shuyi,
>
> Thank you for your input. Yes, I agreed with a phased approach and like to
> move forward fast. :) We did some work internally on DDL utilizing babel
> parser in Calcite. While babel makes Calcite's grammar extensible, at
> first impression it still seems too cumbersome for a project when too
> much extensions are made. It's even challenging to find where the extension
> is needed! It would be certainly better if Calcite can magically support
> Hive QL by just turning on a flag, such as that for MYSQL_5. I can also
> see that this could mean a lot of work on Calcite. Nevertheless, I will
> bring up the discussion over there and to see what their community thinks.
>
> Would mind to share more info about the proposal on DDL that you
> mentioned? We can certainly collaborate on this.
>
> Thanks,
> Xuefu
>
> --
> Sender:Shuyi Chen 
> Sent at:2018 Oct 14 (Sun) 08:30
> Recipient:Xuefu 
> Cc:yanghua1127 ; Fabian Hueske ;
> dev ; user 
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Welcome to the community and thanks for the great proposal, Xuefu! I think
> the proposal can be divided into 2 stages: making Flink to support Hive
> features, and make Hive to work with Flink. I agreed with Timo that on
> starting with a smaller scope, so we can make progress faster. As for [6],
> a proposal for DDL is already in progress, and will come after the unified
> SQL connector API is done. For supporting Hive syntax, we might need to
> work with the Calcite community, and a recent effort called babel (
> https://issues.apache.org/jira/browse/CALCITE-2280) in Calcite might help
> here.
>
> Thanks
> Shuyi
>
> On Wed, Oct 10, 2018 at 8:02 PM Zhang, Xuefu 
> wrote:
> Hi Fabian/Vno,
>
> Thank you very much for your encouragement inquiry. Sorry that I didn't
> see Fabian's email until I read Vino's response just now. (Somehow Fabian's
> went to the spam folder.)
>
> My proposal contains long-term and short-terms goals. Nevertheless, the
> effort will focus on the following areas, including Fabian's list:
>
> 1. Hive metastore connectivity - This covers both read/write access, which
> means 

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

Hi, Timo,
I am very grateful for your feedback, and I am very excited when I hear
that you also consider adding a process function to the TableAPI.

I agree that add support for the Process Function on the Table API, which
is actually part of my proposal Enhancing the functionality of Table API.
In fact, supporting the ProcessFunction means supporting the user-defined
Operator. As you said, A ProcessFunction can implement any logic, including
the user-defined window, which leaves the user with enough freedom and
control. At the same time, Co-PrecessFunction needs to be supported, so we
can implement the logic of User-Defined JOIN through Co-PrecessFunciton. Of
course, Co-PrecessFunciton also needs to introduce the concept of Connect,
and will introduce a new ConnectedTable type on TableAPI.  And I also think
TableAPI also for more event-driven applications.

About processFunction In addition to the timer function, it should be
completely equivalent to flatmapFunction, so maybe we can support map and
flatmap in Table, support processFunction in GroupedTable, because for the
reason of State, the Timer of ProcessFunction can only Apply to KeyedStream.

You are right, ANSI-SQL is difficult to express complex operator logic such
as ProcessFunction, so once we decide to make these enhancements on the
TableAPI, it means that the Flink SQL only includes ANSI-SQL operations,
and the TableAPI' operations is SQL super set. This means that the Flink
High-level API includes the A Query language SQL and A powerfu program
language Table API. In this way, SQL using for those simple ETL user
groups, the TableAPI is for a user group that needs to be customized for
complex logic, and these users can enjoy The benefit of the query
optimizer. Maybe we need more refinement and hard work to support these
functions, but maybe this is a good direction of effort.

Thanks,
Jincheng

Timo Walther  于2018年11月1日周四 下午10:08写道：

> Hi Jincheng,
>
> I was also thinking about introducing a process function for the Table
> API several times. This would allow to define more complex logic (custom
> windows, timers, etc.) embedded into a relational API with schema
> awareness and optimization around the black box. Of course this would
> mean that we diverge with Table API from SQL API, however, it would open
> the Table API also for more event-driven applications.
>
> Maybe it would be possible to define timers and firing logic using Table
> API expressions and UDFs. Within planning this would be treated as a
> special Calc node.
>
> Just some ideas that might be interesting for new use cases.
>
> Regards,
> Timo
>
>
> Am 01.11.18 um 13:12 schrieb Aljoscha Krettek:
> > Hi Jincheng,
> >
> > these points sound very good! Are there any concrete proposals for
> changes? For example a FLIP/design document?
> >
> > See here for FLIPs:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > Best,
> > Aljoscha
> >
> >> On 1. Nov 2018, at 12:51, jincheng sun 
> wrote:
> >>
> >> *I am sorry for the formatting of the email content. I reformat
> >> the **content** as follows---*
> >>
> >> *Hi ALL,*
> >>
> >> With the continuous efforts from the community, the Flink system has
> been
> >> continuously improved, which has attracted more and more users. Flink
> SQL
> >> is a canonical, widely used relational query language. However, there
> are
> >> still some scenarios where Flink SQL failed to meet user needs in terms
> of
> >> functionality and ease of use, such as:
> >>
> >> *1. In terms of functionality*
> >> Iteration, user-defined window, user-defined join, user-defined
> >> GroupReduce, etc. Users cannot express them with SQL;
> >>
> >> *2. In terms of ease of use*
> >>
> >>- Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> >>udf2(), udf3())” can be used to accomplish the same function.,
> with a
> >>map() function returning 100 columns, one has to define or call 100
> UDFs
> >>when using SQL, which is quite involved.
> >>- FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
> be
> >>implemented with “table.join(udtf).select()”. However, it is obvious
> that
> >>dataStream is easier to use than SQL.
> >>
> >> Due to the above two reasons, some users have to use the DataStream API
> or
> >> the DataSet API. But when they do that, they lose the unification of
> batch
> >> and streaming. They will also lose the sophisticated optimizations such
> as
> >> codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> >>
> >> We believe that enhancing the functionality and productivity is vital
> for
> >> the successful adoption of Table API. To this end,  Table API still
> >> requires more efforts from every contributor in the community. We see
> great
> >> opportunity in improving our user’s experience from this work. Any
> feedback
> >> is welcome.
> >>
> >> Regards,
> >>
> >> Jincheng
> >>
> >> jincheng sun  于2018年11月1日周四 

[jira] [Created] (FLINK-10754) Update Jepsen tests to run with activated local recovery

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-10754:
-

 Summary: Update Jepsen tests to run with activated local recovery
 Key: FLINK-10754
 URL: https://issues.apache.org/jira/browse/FLINK-10754
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.7.0


With FLINK-9635 Flink now supports to run properly with local recovery 
activated. We should update the Jepsen tests to run with this feature in order 
to give it more test exposure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10753) Propagate and log snapshotting exceptions

[Alexander Fedulov (JIRA)]

Alexander Fedulov created FLINK-10753:
-

 Summary: Propagate and log snapshotting exceptions
 Key: FLINK-10753
 URL: https://issues.apache.org/jira/browse/FLINK-10753
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.6.2
Reporter: Alexander Fedulov
 Attachments: Screen Shot 2018-11-01 at 16.27.01.png

Upon failure, {{AbstractStreamOperator.snapshotState}} rethrows a new exception 
with the message "{{Could not complete snapshot {} for operator {}.}}" and the 
original exception as the cause. 

While handling the error, {{CheckpointCoordinator.discardCheckpoint}} method 
logs only this  propagated message and not the original cause of the exception.

In addition, {{pendingCheckpoint.abortDeclined()}}, called from the 
{{discardCheckpoint,}}
 reports the failed checkpoint with a misleading message "{{Checkpoint was 
declined (tasks not ready)}}". This message is what will be displayed in the UI 
(see attached).

 Proposition:
 # Log exception at the Task Manager (.snapshotState)
 # Log cause, instead of cause.getMessage() at the JobsManager 
(.dicardCheckpoint)
 # Pass root cause to abortDeclined and propagate a more appropriate message to 
the UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Aljoscha Krettek]

Yes, that makes sense!

> On 1. Nov 2018, at 15:51, jincheng sun  wrote:
> 
> Hi, Aljoscha,
> 
> Thanks for your feedback and suggestions. I think your are right, the
> detailed design/FLIP is very necessary. Before the detailed design or open
> a FLIP, I would like to hear the community's views on Enhancing the
> functionality and productivity of Table API,  to ensure that it worth to
> effort. If most community members agree with my proposal, I will list the
> changes and discuss with all community members. Is that make sense to you?
> 
> Thanks,
> Jincheng
> 
> Aljoscha Krettek  于2018年11月1日周四 下午8:12写道：
> 
>> Hi Jincheng,
>> 
>> these points sound very good! Are there any concrete proposals for
>> changes? For example a FLIP/design document?
>> 
>> See here for FLIPs:
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> 
>> Best,
>> Aljoscha
>> 
>>> On 1. Nov 2018, at 12:51, jincheng sun  wrote:
>>> 
>>> *I am sorry for the formatting of the email content. I reformat
>>> the **content** as follows---*
>>> 
>>> *Hi ALL,*
>>> 
>>> With the continuous efforts from the community, the Flink system has been
>>> continuously improved, which has attracted more and more users. Flink SQL
>>> is a canonical, widely used relational query language. However, there are
>>> still some scenarios where Flink SQL failed to meet user needs in terms
>> of
>>> functionality and ease of use, such as:
>>> 
>>> *1. In terms of functionality*
>>>   Iteration, user-defined window, user-defined join, user-defined
>>> GroupReduce, etc. Users cannot express them with SQL;
>>> 
>>> *2. In terms of ease of use*
>>> 
>>>  - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
>>>  udf2(), udf3())” can be used to accomplish the same function.,
>> with a
>>>  map() function returning 100 columns, one has to define or call 100
>> UDFs
>>>  when using SQL, which is quite involved.
>>>  - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
>>>  implemented with “table.join(udtf).select()”. However, it is obvious
>> that
>>>  dataStream is easier to use than SQL.
>>> 
>>> Due to the above two reasons, some users have to use the DataStream API
>> or
>>> the DataSet API. But when they do that, they lose the unification of
>> batch
>>> and streaming. They will also lose the sophisticated optimizations such
>> as
>>> codegen, aggregate join transpose and multi-stage agg from Flink SQL.
>>> 
>>> We believe that enhancing the functionality and productivity is vital for
>>> the successful adoption of Table API. To this end,  Table API still
>>> requires more efforts from every contributor in the community. We see
>> great
>>> opportunity in improving our user’s experience from this work. Any
>> feedback
>>> is welcome.
>>> 
>>> Regards,
>>> 
>>> Jincheng
>>> 
>>> jincheng sun  于2018年11月1日周四 下午5:07写道：
>>> 
 Hi all,
 
 With the continuous efforts from the community, the Flink system has
>> been
 continuously improved, which has attracted more and more users. Flink
>> SQL
 is a canonical, widely used relational query language. However, there
>> are
 still some scenarios where Flink SQL failed to meet user needs in terms
>> of
 functionality and ease of use, such as:
 
 
  -
 
  In terms of functionality
 
 Iteration, user-defined window, user-defined join, user-defined
 GroupReduce, etc. Users cannot express them with SQL;
 
  -
 
  In terms of ease of use
  -
 
 Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
 udf2(), udf3())” can be used to accomplish the same function.,
>> with a
 map() function returning 100 columns, one has to define or call
>> 100 UDFs
 when using SQL, which is quite involved.
 -
 
 FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
 be implemented with “table.join(udtf).select()”. However, it is
>> obvious
 that datastream is easier to use than SQL.
 
 
 Due to the above two reasons, some users have to use the DataStream API
>> or
 the DataSet API. But when they do that, they lose the unification of
>> batch
 and streaming. They will also lose the sophisticated optimizations such
>> as
 codegen, aggregate join transpose  and multi-stage agg from Flink SQL.
 
 We believe that enhancing the functionality and productivity is vital
>> for
 the successful adoption of Table API. To this end,  Table API still
 requires more efforts from every contributor in the community. We see
>> great
 opportunity in improving our user’s experience from this work. Any
>> feedback
 is welcome.
 
 Regards,
 
 Jincheng
 
 
>> 
>> 

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

Hi, Aljoscha,

Thanks for your feedback and suggestions. I think your are right, the
detailed design/FLIP is very necessary. Before the detailed design or open
a FLIP, I would like to hear the community's views on Enhancing the
functionality and productivity of Table API,  to ensure that it worth to
effort. If most community members agree with my proposal, I will list the
changes and discuss with all community members. Is that make sense to you?

Thanks,
Jincheng

Aljoscha Krettek  于2018年11月1日周四 下午8:12写道：

> Hi Jincheng,
>
> these points sound very good! Are there any concrete proposals for
> changes? For example a FLIP/design document?
>
> See here for FLIPs:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> Best,
> Aljoscha
>
> > On 1. Nov 2018, at 12:51, jincheng sun  wrote:
> >
> > *I am sorry for the formatting of the email content. I reformat
> > the **content** as follows---*
> >
> > *Hi ALL,*
> >
> > With the continuous efforts from the community, the Flink system has been
> > continuously improved, which has attracted more and more users. Flink SQL
> > is a canonical, widely used relational query language. However, there are
> > still some scenarios where Flink SQL failed to meet user needs in terms
> of
> > functionality and ease of use, such as:
> >
> > *1. In terms of functionality*
> >Iteration, user-defined window, user-defined join, user-defined
> > GroupReduce, etc. Users cannot express them with SQL;
> >
> > *2. In terms of ease of use*
> >
> >   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> >   udf2(), udf3())” can be used to accomplish the same function.,
> with a
> >   map() function returning 100 columns, one has to define or call 100
> UDFs
> >   when using SQL, which is quite involved.
> >   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
> >   implemented with “table.join(udtf).select()”. However, it is obvious
> that
> >   dataStream is easier to use than SQL.
> >
> > Due to the above two reasons, some users have to use the DataStream API
> or
> > the DataSet API. But when they do that, they lose the unification of
> batch
> > and streaming. They will also lose the sophisticated optimizations such
> as
> > codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> >
> > We believe that enhancing the functionality and productivity is vital for
> > the successful adoption of Table API. To this end,  Table API still
> > requires more efforts from every contributor in the community. We see
> great
> > opportunity in improving our user’s experience from this work. Any
> feedback
> > is welcome.
> >
> > Regards,
> >
> > Jincheng
> >
> > jincheng sun  于2018年11月1日周四 下午5:07写道：
> >
> >> Hi all,
> >>
> >> With the continuous efforts from the community, the Flink system has
> been
> >> continuously improved, which has attracted more and more users. Flink
> SQL
> >> is a canonical, widely used relational query language. However, there
> are
> >> still some scenarios where Flink SQL failed to meet user needs in terms
> of
> >> functionality and ease of use, such as:
> >>
> >>
> >>   -
> >>
> >>   In terms of functionality
> >>
> >> Iteration, user-defined window, user-defined join, user-defined
> >> GroupReduce, etc. Users cannot express them with SQL;
> >>
> >>   -
> >>
> >>   In terms of ease of use
> >>   -
> >>
> >>  Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
> >>  udf2(), udf3())” can be used to accomplish the same function.,
> with a
> >>  map() function returning 100 columns, one has to define or call
> 100 UDFs
> >>  when using SQL, which is quite involved.
> >>  -
> >>
> >>  FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
> >>  be implemented with “table.join(udtf).select()”. However, it is
> obvious
> >>  that datastream is easier to use than SQL.
> >>
> >>
> >> Due to the above two reasons, some users have to use the DataStream API
> or
> >> the DataSet API. But when they do that, they lose the unification of
> batch
> >> and streaming. They will also lose the sophisticated optimizations such
> as
> >> codegen, aggregate join transpose  and multi-stage agg from Flink SQL.
> >>
> >> We believe that enhancing the functionality and productivity is vital
> for
> >> the successful adoption of Table API. To this end,  Table API still
> >> requires more efforts from every contributor in the community. We see
> great
> >> opportunity in improving our user’s experience from this work. Any
> feedback
> >> is welcome.
> >>
> >> Regards,
> >>
> >> Jincheng
> >>
> >>
>
>

[jira] [Created] (FLINK-10752) Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-10752:


 Summary: Result of 
AbstractYarnClusterDescriptor#validateClusterResources is ignored
 Key: FLINK-10752
 URL: https://issues.apache.org/jira/browse/FLINK-10752
 Project: Flink
  Issue Type: Bug
  Components: YARN
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.7.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10751) Checkpoints should be retained when job reaches suspended state

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-10751:
---

 Summary: Checkpoints should be retained when job reaches suspended 
state
 Key: FLINK-10751
 URL: https://issues.apache.org/jira/browse/FLINK-10751
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.6.2
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi


{{CheckpointProperties}} define in which terminal job status a checkpoint 
should be disposed.

I've noticed that the properties for {{CHECKPOINT_NEVER_RETAINED}}, 
{{CHECKPOINT_RETAINED_ON_FAILURE}} prescribe checkpoint disposal in (locally) 
terminal job status {{SUSPENDED}}.

Since a job reaches the {{SUSPENDED}} state when its {{JobMaster}} looses 
leadership, this would result in the checkpoint to be cleaned up and not being 
available for recovery by the new leader. Therefore, we should rather retain 
checkpoints when reaching job status {{SUSPENDED}}.

*BUT:* Because we special case this terminal state in the only highly available 
{{CompletedCheckpointStore}} implementation (see 
[ZooKeeperCompletedCheckpointStore|https://github.com/apache/flink/blob/e7ac3ba/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L315])
 and don't use regular checkpoint disposal, this issue has not surfaced yet.

I think we should proactively fix the properties to indicate to retain 
checkpoints in {{SUSPENDED}} state. We might actually completely remove this 
case since with this change, all properties will indicate to retain on 
suspension.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Timo Walther]

Hi Jincheng,

I was also thinking about introducing a process function for the Table 
API several times. This would allow to define more complex logic (custom 
windows, timers, etc.) embedded into a relational API with schema 
awareness and optimization around the black box. Of course this would 
mean that we diverge with Table API from SQL API, however, it would open 
the Table API also for more event-driven applications.


Maybe it would be possible to define timers and firing logic using Table 
API expressions and UDFs. Within planning this would be treated as a 
special Calc node.


Just some ideas that might be interesting for new use cases.

Regards,
Timo


Am 01.11.18 um 13:12 schrieb Aljoscha Krettek:

Hi Jincheng,

these points sound very good! Are there any concrete proposals for changes? For 
example a FLIP/design document?

See here for FLIPs: 
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Aljoscha


On 1. Nov 2018, at 12:51, jincheng sun  wrote:

*I am sorry for the formatting of the email content. I reformat
the **content** as follows---*

*Hi ALL,*

With the continuous efforts from the community, the Flink system has been
continuously improved, which has attracted more and more users. Flink SQL
is a canonical, widely used relational query language. However, there are
still some scenarios where Flink SQL failed to meet user needs in terms of
functionality and ease of use, such as:

*1. In terms of functionality*
Iteration, user-defined window, user-defined join, user-defined
GroupReduce, etc. Users cannot express them with SQL;

*2. In terms of ease of use*

   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
   udf2(), udf3())” can be used to accomplish the same function., with a
   map() function returning 100 columns, one has to define or call 100 UDFs
   when using SQL, which is quite involved.
   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
   implemented with “table.join(udtf).select()”. However, it is obvious that
   dataStream is easier to use than SQL.

Due to the above two reasons, some users have to use the DataStream API or
the DataSet API. But when they do that, they lose the unification of batch
and streaming. They will also lose the sophisticated optimizations such as
codegen, aggregate join transpose and multi-stage agg from Flink SQL.

We believe that enhancing the functionality and productivity is vital for
the successful adoption of Table API. To this end,  Table API still
requires more efforts from every contributor in the community. We see great
opportunity in improving our user’s experience from this work. Any feedback
is welcome.

Regards,

Jincheng

jincheng sun  于2018年11月1日周四 下午5:07写道：


Hi all,

With the continuous efforts from the community, the Flink system has been
continuously improved, which has attracted more and more users. Flink SQL
is a canonical, widely used relational query language. However, there are
still some scenarios where Flink SQL failed to meet user needs in terms of
functionality and ease of use, such as:


   -

   In terms of functionality

Iteration, user-defined window, user-defined join, user-defined
GroupReduce, etc. Users cannot express them with SQL;

   -

   In terms of ease of use
   -

  Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
  udf2(), udf3())” can be used to accomplish the same function., with a
  map() function returning 100 columns, one has to define or call 100 UDFs
  when using SQL, which is quite involved.
  -

  FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
  be implemented with “table.join(udtf).select()”. However, it is obvious
  that datastream is easier to use than SQL.


Due to the above two reasons, some users have to use the DataStream API or
the DataSet API. But when they do that, they lose the unification of batch
and streaming. They will also lose the sophisticated optimizations such as
codegen, aggregate join transpose  and multi-stage agg from Flink SQL.

We believe that enhancing the functionality and productivity is vital for
the successful adoption of Table API. To this end,  Table API still
requires more efforts from every contributor in the community. We see great
opportunity in improving our user’s experience from this work. Any feedback
is welcome.

Regards,

Jincheng

[jira] [Created] (FLINK-10750) SocketClientSinkTest.testRetry fails on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-10750:
-

 Summary: SocketClientSinkTest.testRetry fails on Travis
 Key: FLINK-10750
 URL: https://issues.apache.org/jira/browse/FLINK-10750
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
 Fix For: 1.7.0


The {{SocketClientSinkTest.testRetry}} fails on Travis because of a 
{{BindException}}: https://api.travis-ci.org/v3/job/448907069/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[Aljoscha Krettek]

Hi Jincheng,

these points sound very good! Are there any concrete proposals for changes? For 
example a FLIP/design document?

See here for FLIPs: 
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Aljoscha 

> On 1. Nov 2018, at 12:51, jincheng sun  wrote:
> 
> *I am sorry for the formatting of the email content. I reformat
> the **content** as follows---*
> 
> *Hi ALL,*
> 
> With the continuous efforts from the community, the Flink system has been
> continuously improved, which has attracted more and more users. Flink SQL
> is a canonical, widely used relational query language. However, there are
> still some scenarios where Flink SQL failed to meet user needs in terms of
> functionality and ease of use, such as:
> 
> *1. In terms of functionality*
>Iteration, user-defined window, user-defined join, user-defined
> GroupReduce, etc. Users cannot express them with SQL;
> 
> *2. In terms of ease of use*
> 
>   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
>   udf2(), udf3())” can be used to accomplish the same function., with a
>   map() function returning 100 columns, one has to define or call 100 UDFs
>   when using SQL, which is quite involved.
>   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
>   implemented with “table.join(udtf).select()”. However, it is obvious that
>   dataStream is easier to use than SQL.
> 
> Due to the above two reasons, some users have to use the DataStream API or
> the DataSet API. But when they do that, they lose the unification of batch
> and streaming. They will also lose the sophisticated optimizations such as
> codegen, aggregate join transpose and multi-stage agg from Flink SQL.
> 
> We believe that enhancing the functionality and productivity is vital for
> the successful adoption of Table API. To this end,  Table API still
> requires more efforts from every contributor in the community. We see great
> opportunity in improving our user’s experience from this work. Any feedback
> is welcome.
> 
> Regards,
> 
> Jincheng
> 
> jincheng sun  于2018年11月1日周四 下午5:07写道：
> 
>> Hi all,
>> 
>> With the continuous efforts from the community, the Flink system has been
>> continuously improved, which has attracted more and more users. Flink SQL
>> is a canonical, widely used relational query language. However, there are
>> still some scenarios where Flink SQL failed to meet user needs in terms of
>> functionality and ease of use, such as:
>> 
>> 
>>   -
>> 
>>   In terms of functionality
>> 
>> Iteration, user-defined window, user-defined join, user-defined
>> GroupReduce, etc. Users cannot express them with SQL;
>> 
>>   -
>> 
>>   In terms of ease of use
>>   -
>> 
>>  Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
>>  udf2(), udf3())” can be used to accomplish the same function., with 
>> a
>>  map() function returning 100 columns, one has to define or call 100 UDFs
>>  when using SQL, which is quite involved.
>>  -
>> 
>>  FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
>>  be implemented with “table.join(udtf).select()”. However, it is obvious
>>  that datastream is easier to use than SQL.
>> 
>> 
>> Due to the above two reasons, some users have to use the DataStream API or
>> the DataSet API. But when they do that, they lose the unification of batch
>> and streaming. They will also lose the sophisticated optimizations such as
>> codegen, aggregate join transpose  and multi-stage agg from Flink SQL.
>> 
>> We believe that enhancing the functionality and productivity is vital for
>> the successful adoption of Table API. To this end,  Table API still
>> requires more efforts from every contributor in the community. We see great
>> opportunity in improving our user’s experience from this work. Any feedback
>> is welcome.
>> 
>> Regards,
>> 
>> Jincheng
>> 
>> 

[ANNOUNCE] Apache Flink-shaded 5.0 released

[Chesnay Schepler]

The Apache Flink community is pleased to announce the release of Apache 
Flink-shaded 5.0.


The flink-shaded project contains a number of shaded dependencies for 
Apache Flink.



This release adds jackson-dataformat-csv to the flink-shaded-jackson 
module and adds a new flink-shaded-asm6 module required for Java 
9+/Scala 2.12.



For more information about flink-shaded, see 
https://issues.apache.org/jira/browse/FLINK-6529.


The release is available for download at:

   https://flink.apache.org/downloads.html

PS: flink-shaded 5.0 was already released 2 weeks ago but I forgot to 
send out the announcement mail.

[jira] [Created] (FLINK-10749) GraphOperationsITCase deadlocks on travis

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-10749:


 Summary: GraphOperationsITCase deadlocks on travis
 Key: FLINK-10749
 URL: https://issues.apache.org/jira/browse/FLINK-10749
 Project: Flink
  Issue Type: Bug
  Components: Gelly
Affects Versions: 1.7.0
Reporter: Chesnay Schepler


https://travis-ci.org/apache/flink/jobs/449173144



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

*I am sorry for the formatting of the email content. I reformat
the **content** as follows---*

*Hi ALL,*

With the continuous efforts from the community, the Flink system has been
continuously improved, which has attracted more and more users. Flink SQL
is a canonical, widely used relational query language. However, there are
still some scenarios where Flink SQL failed to meet user needs in terms of
functionality and ease of use, such as:

*1. In terms of functionality*
Iteration, user-defined window, user-defined join, user-defined
GroupReduce, etc. Users cannot express them with SQL;

*2. In terms of ease of use*

   - Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
   udf2(), udf3())” can be used to accomplish the same function., with a
   map() function returning 100 columns, one has to define or call 100 UDFs
   when using SQL, which is quite involved.
   - FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
   implemented with “table.join(udtf).select()”. However, it is obvious that
   dataStream is easier to use than SQL.

Due to the above two reasons, some users have to use the DataStream API or
the DataSet API. But when they do that, they lose the unification of batch
and streaming. They will also lose the sophisticated optimizations such as
codegen, aggregate join transpose and multi-stage agg from Flink SQL.

We believe that enhancing the functionality and productivity is vital for
the successful adoption of Table API. To this end,  Table API still
requires more efforts from every contributor in the community. We see great
opportunity in improving our user’s experience from this work. Any feedback
is welcome.

Regards,

Jincheng

jincheng sun  于2018年11月1日周四 下午5:07写道：

> Hi all,
>
> With the continuous efforts from the community, the Flink system has been
> continuously improved, which has attracted more and more users. Flink SQL
> is a canonical, widely used relational query language. However, there are
> still some scenarios where Flink SQL failed to meet user needs in terms of
> functionality and ease of use, such as:
>
>
>-
>
>In terms of functionality
>
> Iteration, user-defined window, user-defined join, user-defined
> GroupReduce, etc. Users cannot express them with SQL;
>
>-
>
>In terms of ease of use
>-
>
>   Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
>   udf2(), udf3())” can be used to accomplish the same function., with 
> a
>   map() function returning 100 columns, one has to define or call 100 UDFs
>   when using SQL, which is quite involved.
>   -
>
>   FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can
>   be implemented with “table.join(udtf).select()”. However, it is obvious
>   that datastream is easier to use than SQL.
>
>
> Due to the above two reasons, some users have to use the DataStream API or
> the DataSet API. But when they do that, they lose the unification of batch
> and streaming. They will also lose the sophisticated optimizations such as
> codegen, aggregate join transpose  and multi-stage agg from Flink SQL.
>
> We believe that enhancing the functionality and productivity is vital for
> the successful adoption of Table API. To this end,  Table API still
> requires more efforts from every contributor in the community. We see great
> opportunity in improving our user’s experience from this work. Any feedback
> is welcome.
>
> Regards,
>
> Jincheng
>
>

[jira] [Created] (FLINK-10747) CoGroupConnectedComponentsITCase deadlocked on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-10747:
-

 Summary: CoGroupConnectedComponentsITCase deadlocked on Travis
 Key: FLINK-10747
 URL: https://issues.apache.org/jira/browse/FLINK-10747
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
 Fix For: 1.7.0


The {{CoGroupConnectedComponentsITCase}} deadlocks on Travis: 
https://api.travis-ci.org/v3/job/449230962/log.txt.

It looks as if the iteration gets stuck and waits 
{{SuperstepKickoffLatch.awaitStartOfSuperstepOrTermination}} without ever 
getting a proper notification. This might indicate a serious bug in our 
iteration implementation. This could also be related to FLINK-10741.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10748) Jobmanager in HA setup redirects to ip address instead of hostname

[Jeroen Steggink (JIRA)]

Jeroen Steggink created FLINK-10748:
---

 Summary: Jobmanager in HA setup redirects to ip address instead of 
hostname
 Key: FLINK-10748
 URL: https://issues.apache.org/jira/browse/FLINK-10748
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 1.6.2, 1.5.5, 1.4.2, 1.3.3
Reporter: Jeroen Steggink


In a HA Jobmanager setup, connecting to a follower redirects (HTTP/1.1 307 
Temporary Redirect) to the leader. However, it redirects  to an ip address 
instead of the hostname. This can result in a situation where a server is not 
resolved because the routing is done based on hostname.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10746) Need to replace transfer.sh for Travis log upload because it shuts down

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-10746:
-

 Summary: Need to replace transfer.sh for Travis log upload because 
it shuts down
 Key: FLINK-10746
 URL: https://issues.apache.org/jira/browse/FLINK-10746
 Project: Flink
  Issue Type: Task
  Components: Build System
Affects Versions: 1.7.0
Reporter: Till Rohrmann
 Fix For: 1.8.0


We need to replace {{transfer.sh}} as the destination for our Travis log upload 
because it is about to shut down (see https://transfer.sh/).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Sharing state between subtasks

[Till Rohrmann]

Hi Thomas,

using Akka directly would further manifest our dependency on Scala in
flink-runtime. This is something we are currently trying to get rid of. For
that purpose we have added the RpcService abstraction which encapsulates
all Akka specific logic. We hope that we can soon get rid of the Scala
dependency in flink-runtime by using a special class loader only for
loading the AkkaRpcService implementation.

I think the easiest way to sync the task information is actually going
through the JobMaster because the subtasks don't know on which other TMs
the other subtasks run. Otherwise, we would need to have some TM detection
mechanism between TMs. If you choose this way, then you should be able to
use the RpcService by extending the JobMasterGateway by additional RPCs.

Cheers,
Till

On Wed, Oct 31, 2018 at 6:52 PM Thomas Weise  wrote:

> Hi,
>
> We are planning to work on the Kinesis consumer in the following order:
>
> 1. Add per shard watermarking:
> https://issues.apache.org/jira/browse/FLINK-5697 - this will be code we
> already use internally; I will open a PR to add it to the Flink Kinesis
> consumer
> 2. Exchange of per subtask watermarks between all subtasks of one or
> multiple sources
> 3. Implement queue approach described in Jamie's document in to utilize 1.)
> and 2.) to align the shard consumers WRT event time
>
> There was some discussion regarding the mechanism to share the watermarks
> between subtasks. If there is something that can be re-used it would be
> great. Otherwise I'm going to further investigate the Akka or JGroups
> route. Regarding Akka, since it is used within Flink already, is there an
> abstraction that you would recommend to consider to avoid direct
> dependency?
>
> Thanks,
> Thomas
>
>
>
> On Thu, Oct 18, 2018 at 8:34 PM Zhijiang(wangzhijiang999)
>  wrote:
>
> > Not yet. We only have some initial thoughts and have not worked on it
> yet.
> > We will update the progress in this discussion if have.
> >
> > Best,
> > Zhijiang
> > --
> > 发件人：Aljoscha Krettek 
> > 发送时间：2018年10月18日(星期四) 17:53
> > 收件人：dev ; Zhijiang(wangzhijiang999) <
> > wangzhijiang...@aliyun.com>
> > 抄 送：Till Rohrmann 
> > 主 题：Re: Sharing state between subtasks
> >
> > Hi Zhijiang,
> >
> > do you already have working code or a design doc for the second approach?
> >
> > Best,
> > Aljoscha
> >
> > > On 18. Oct 2018, at 08:42, Zhijiang(wangzhijiang999) <
> > wangzhijiang...@aliyun.com.INVALID> wrote:
> > >
> > > Just noticed this discussion from @Till Rohrmann's weekly community
> > update and I want to share some thoughts from our experiences.
> > >
> > > We also encountered the source consuption skew issue before, and we are
> > focused on improving this by two possible ways.
> > >
> > > 1. Control the read strategy by the downstream side. In detail, every
> > input channel in downstream task corresponds to the consumption of one
> > upstream source task, and we will tag each input channel with watermark
> to
> > find the lowest channel to read in high priority. In essence, we actually
> > rely on the mechanism of backpressure. If the channel with highest
> > timestamp is not read by downstream task for a while, it will block the
> > corresponding source task to read when the buffers are exhausted. It is
> no
> > need to change the source interface in this way, but there are two major
> > concerns: first it will affect the barier alignment resulting in
> checkpoint
> > delayed or expired. Second it can not confirm source consumption
> alignment
> > very precisely, and it is just a best effort way. So we gave up this way
> > finally.
> > >
> > > 2. Add the new component of SourceCoordinator to coordinate the source
> > consumption distributedly. For example we can start this componnet in the
> > JobManager like the current role of CheckpointCoordinator. Then every
> > source task would commnicate with JobManager via current RPC mechanism,
> > maybe we can rely on the heartbeat message to attach the consumption
> > progress as the payloads. The JobManagerwill accumulator or state all the
> > reported progress and then give responses for different source tasks. We
> > can define a protocol for indicating the fast soruce task to sleep for
> > specific time for example. To do so, the coordinator has the global
> > informations to give the proper decision for individuals, so it seems
> more
> > precise. And it will not affect the barrier alignment, because the
> sleeping
> > fast source can release the lock to emit barrier as normal. The only
> > concern is the changes for source interface and may affect all related
> > source implementations.
> > >
> > > Currently we prefer to the second way to implement and will refer to
> > other good points above. :)
> > >
> > > Best,
> > > Zhijiang
> > > --
> > > 发件人：Jamie Grier 
> > > 发送时间：2018年10月17日(星期三) 03:28
> > > 收件人：dev 
> > > 主 题：Re: 

Re: Flip23

[Till Rohrmann]

Thanks for sharing the code with the community Boris!

Cheers,
Till

On Thu, Nov 1, 2018 at 10:29 AM Boris Lublinsky <
boris.lublin...@lightbend.com> wrote:

> For those who want to see an actual code, it is here:
> https://github.com/FlinkML/flink-modelServer <
> https://github.com/FlinkML/flink-modelServer> for simple model serving
> And https://github.com/FlinkML/flink-modelServer <
> https://github.com/FlinkML/flink-modelServer> for speculative one
>
> Boris Lublinsky
> FDP Architect
> boris.lublin...@lightbend.com
> https://www.lightbend.com/
>
>

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Piotr Nowojski]

Hi,

Thanks Aljoscha for starting this, it’s blocking quite a lot of other possible 
improvements. I have one proposal. Instead of having a method:

boolean advance() throws IOException;

I would replace it with

/* 
 * Return a future, which when completed means that source has more data and 
getNext() will not block.
 * If you wish to use benefits of non blocking connectors, please implement 
this method appropriately.
 */
default CompletableFuture isBlocked() {
return CompletableFuture.completedFuture(null);
}

And rename `getCurrent()` to `getNext()`.

Couple of arguments:
1. I don’t understand the division of work between `advance()` and 
`getCurrent()`. What should be done in which, especially for connectors that 
handle records in batches (like Kafka) and when should you call `advance` and 
when `getCurrent()`. 
2. Replacing `boolean` with `CompletableFuture` will allow us in the future 
to have asynchronous/non blocking connectors and more efficiently handle large 
number of blocked threads, without busy waiting. While at the same time it 
doesn’t add much complexity, since naive connector implementations can be 
always blocking.
3. This also would allow us to use a fixed size thread pool of task executors, 
instead of one thread per task.

Piotrek

> On 31 Oct 2018, at 17:22, Aljoscha Krettek  wrote:
> 
> Hi All,
> 
> In order to finally get the ball rolling on the new source interface that we 
> have discussed for so long I finally created a FLIP: 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> 
> I cc'ed Thomas and Jamie because of the ongoing work/discussion about adding 
> per-partition watermark support to the Kinesis source and because this would 
> enable generic implementation of event-time alignment for all sources. Maybe 
> we need another FLIP for the event-time alignment part, especially the part 
> about information sharing between operations (I'm not calling it state 
> sharing because state has a special meaning in Flink).
> 
> Please discuss away!
> 
> Aljoscha
> 
> 

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

[Yadong Xie]

For your interest, here are some snapshots of Flink web UI (with angular
7.0) we recently reconstructed.
[image: snapshot.001.jpeg]
https://drive.google.com/file/d/1p7rcmgHi3ZJpjt7h1H97VIrIhNkiBs1j/view?usp=sharing

We choose Angular 7.0 due to the following reasons:

   1. It is easy to migrate the current codebase to Angular 7.0 following
   the official guide.
   2. It is easy for current Web UI developers who are experienced with
   Angular 1.x to get familiar with 7.0 reading the official doc.
   3. Migrating other than rewriting is helpful to ensure its functional
   integrity.
   4. There are over 1 million Angular developers now (2.0 and above, still
   increasing rapidly), which means there are potentially more developers
   willing to join the community.
   5. No bower or complex node / webpack configuration needed (only angular
   cli).

We split Angular 1.x and 7.0 versions into two folders based on the
following consideration:

   1. 1.x version could be the default page because there is no breakchange
   for both users and developers.
   2. Make the upgrade progressive. Users can switch between two versions
   by clicking the button on the top right corner.
   3. Reduce maintenance costs and keep the performance of web without
   bootstrap both 1.x and 7.0 versions in the same page.
   4. Won't interrupt the current development process. The code can still
   be submitted to 1.x version folder, we could help to sync all features and
   bug fix to 7.0 version if needed.
   5. When the 7.0 version is ready and stable, it is easy to change
   default page to it, and remove the old version.

[image: test.jpg]

https://drive.google.com/file/d/1v7U5upP0fSepVl-E_46k-Qfj9uVfrGZf/view?usp=sharing

Here is the brief list of what we have done:

   1. Create a new folder named next-web-dashboard in flink-runtime-web.
   2. Migrate all Angular 1.x controllers, services, config etc one by one
   into 7.0 version with ngMigration.
   3. Modify frontend build scripts in order to package two versions
   together.
   4. All works listed above are limited in the flink-runtime-web folder.
   One folder was added and no file or folder was deleted.

The new version of the Web UI is running stable for several months in
Alibaba. We are happy to contribute it to the Flink community if everyone
like the work.

Best
Yadong

On Wed, Oct 31, 2018 at 5:48 PM Till Rohrmann  wrote:

> I agree with Chesnay that the web UI is already fairly isolated in Flink's
> code base (see flink-runtime-web module). Moreover, imo it is very
> important that Flink comes with a nice web UI which works out of the box
> because it lowers the entrance hurdle massively. What one could argue is to
> make it easier to plug in your own web UI if you want to develop it. But I
> would see this as an orthogonal issue.
>
> I would actually argue against Robert's proposal to make a grand rewrite if
> this is by any means possible. Experience shows that these kind of efforts
> are usually set up to fail or take super long. If possible, then I would
> rather prefer to convert one component at a time to React. But I'm not an
> expert here and cannot tell whether this is possible or not.
>
> What we would need is active help from the community to review these
> changes. I think that is the crucial piece to make such an effort work or
> fail.
>
> Cheers,
> Till
>
> On Wed, Oct 31, 2018 at 10:14 AM Chesnay Schepler 
> wrote:
>
> > The WebUI goes entirely through the REST API.
> >
> > I don't see how moving it out would make it easier to develop/contribute.
> >
> > The UI is already encapsulated in it's own module (flink-runtime-web)
> > and changes can be made to it independently.
> > Historically the issue was simply that contributions to the UI were not
> > reviewed in a timely manner /  not at all, which mind you
> > applies to virtually all components. This also won't change by moving it
> > out of Flink.
> >
> > Having it in Flink also guarantees that there is an easily-discoverable
> > version of the UI that is compatible with your Flink distribution.
> >
> > On 31.10.2018 09:53, dyana.rose wrote:
> > > Re: who's using the web ui
> > >
> > > Though many mature solutions, with a fair amount of time/resources
> > available are likely running their own front ends, for teams like mine
> > which are smaller and aren't focused solely on working with Flink, having
> > the web ui available removes a large barrier to getting up and running
> > quickly.
> > >
> > > But, with the web UI being in core Flink, it does make it a bit more of
> > a chore to contribute.
> > >
> > > I'm unaware of how necessary it is for the UI to deploy with Flink? Is
> > it making any calls that aren't through the REST API (and if so, can
> those
> > endpoints be added to the REST API)?
> > >
> > > In general I'd support moving it to its own package, making it easier
> to
> > develop with your standard UI dev tools. This also allows the web UI to
> be
> > developed and released 

Flip23

[Boris Lublinsky]

For those who want to see an actual code, it is here:
https://github.com/FlinkML/flink-modelServer 
 for simple model serving
And https://github.com/FlinkML/flink-modelServer 
 for speculative one

Boris Lublinsky
FDP Architect
boris.lublin...@lightbend.com
https://www.lightbend.com/

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

[Yadong Xie]

Thanks to Fabian for starting this discussion!
I liked the discussion. Here are some opinions I have:
[1] Till mentioned that "I think our web UI technology stack is quite dusty
by now and it would be beneficial to think about its technological future."
I totally agree with that we should upgrade the web UI technology stack. I
have been working on Flink Web UI for quite a while. We recently
reconstructed it to the latest version of Angular (7.0) in Alibaba. and we
would love to contribute the entire work to the Flink community. I will
show some snapshots and provide more details later in this discussion
thread.
[2] Fabian mentioned that "From my experience the choice between Angular
2-7 or React is subjective, you can get things done with both."
I totally agree with this. But to my experience, upgrade Angular from 1 to
2-7 is relatively easier and less work than completely rewrite the entire
web UI via React. And since we have already completed angular 1 to angular
7 upgrading, I hope we can keep the web UI to angular.

Best
Yadong

On Wed, Oct 31, 2018 at 5:48 PM Till Rohrmann  wrote:

> I agree with Chesnay that the web UI is already fairly isolated in Flink's
> code base (see flink-runtime-web module). Moreover, imo it is very
> important that Flink comes with a nice web UI which works out of the box
> because it lowers the entrance hurdle massively. What one could argue is to
> make it easier to plug in your own web UI if you want to develop it. But I
> would see this as an orthogonal issue.
>
> I would actually argue against Robert's proposal to make a grand rewrite if
> this is by any means possible. Experience shows that these kind of efforts
> are usually set up to fail or take super long. If possible, then I would
> rather prefer to convert one component at a time to React. But I'm not an
> expert here and cannot tell whether this is possible or not.
>
> What we would need is active help from the community to review these
> changes. I think that is the crucial piece to make such an effort work or
> fail.
>
> Cheers,
> Till
>
> On Wed, Oct 31, 2018 at 10:14 AM Chesnay Schepler 
> wrote:
>
> > The WebUI goes entirely through the REST API.
> >
> > I don't see how moving it out would make it easier to develop/contribute.
> >
> > The UI is already encapsulated in it's own module (flink-runtime-web)
> > and changes can be made to it independently.
> > Historically the issue was simply that contributions to the UI were not
> > reviewed in a timely manner /  not at all, which mind you
> > applies to virtually all components. This also won't change by moving it
> > out of Flink.
> >
> > Having it in Flink also guarantees that there is an easily-discoverable
> > version of the UI that is compatible with your Flink distribution.
> >
> > On 31.10.2018 09:53, dyana.rose wrote:
> > > Re: who's using the web ui
> > >
> > > Though many mature solutions, with a fair amount of time/resources
> > available are likely running their own front ends, for teams like mine
> > which are smaller and aren't focused solely on working with Flink, having
> > the web ui available removes a large barrier to getting up and running
> > quickly.
> > >
> > > But, with the web UI being in core Flink, it does make it a bit more of
> > a chore to contribute.
> > >
> > > I'm unaware of how necessary it is for the UI to deploy with Flink? Is
> > it making any calls that aren't through the REST API (and if so, can
> those
> > endpoints be added to the REST API)?
> > >
> > > In general I'd support moving it to its own package, making it easier
> to
> > develop with your standard UI dev tools. This also allows the web UI to
> be
> > developed and released independently to core Flink.
> > >
> > > Dyana
> > >
> > > On 2018/10/31 07:47:50, Fabian Wollert  wrote:
> > >> Hi Till, I basically agree with all your points. i would stress the
> > >> "dustiness" of the current architecture: the package manager used
> > (bower)
> > >> is deprecated since a long time, the chance for the builds of the
> flink
> > web
> > >> dashboard not working anymore is increasing every day.
> > >>
> > >> About the knowledge in the community: Two days is not a lot of time,
> but
> > >> interest in this topic seems to be minor anyways. Is someone using the
> > >> Flink Web Dashboard at all, or is everyone running their own custom
> > >> solutions? Because if there is no interest in using the Web UI AND no
> > one
> > >> interested in developing, there would be no need to package this as
> > part of
> > >> the official Flink package, but rather develop an independent solution
> > (I'm
> > >> not voting for this right now, just putting it out), if at all. The
> > >> official package could then just ship with the API, which other
> > solutions
> > >> can build upon. This solution could be from an agile point of view
> also
> > the
> > >> best (enhanced testing, independent and more effective dev workflow,
> > etc.),
> > >> but is bad for the usage of the Flink UI, because 

回复：[DISCUSS] Proposal of external shuffle service

[zhijiang]

Thanks for the efficient response till!

Thanks sunjin for the good feedbacks, we will further confirm with the comments 
then! :)
--
发件人：Jin Sun 
发送时间：2018年11月1日(星期四) 06:42
收件人：dev 
抄　送：Zhijiang(wangzhijiang999) ; Nico Kruber 
; Piotr Nowojski ; Stephan 
Ewen 
主　题：Re: [DISCUSS] Proposal of external shuffle service

Thanks Zhijiang for the proposal. I like the idea of external shuffle service, 
have left some comments on the document. 

> On Oct 31, 2018, at 2:26 AM, Till Rohrmann  wrote:
> 
> Thanks for the update Zhijiang! The community is currently quite busy with
> the next Flink release. I hope that we can finish the release in two weeks.
> After that people will become more responsive again.
> 
> Cheers,
> Till
> 
> On Wed, Oct 31, 2018 at 7:49 AM zhijiang  wrote:
> 
>> I already created the umbrella jira [1] for this improvement, and attched
>> the design doc [2] in this jira.
>> 
>> Welcome for further discussion about the details.
>> 
>> [1] https://issues.apache.org/jira/browse/FLINK-10653
>> [2]
>> https://docs.google.com/document/d/1Jb0Mf46ace-6cLRQxJzo6VNQQVxn3hwf9Zqmv5pcb34/edit?usp=sharing
>> 
>> 
>> 
>> Best,
>> Zhijiang
>> 
>> --
>> 发件人：Zhijiang(wangzhijiang999) 
>> 发送时间：2018年9月11日(星期二) 15:21
>> 收件人：dev 
>> 抄 送：dev 
>> 主 题：回复：[DISCUSS] Proposal of external shuffle service
>> 
>> Many thanks Till!
>> 
>> 
>> I would create a JIRA for this feature and design a document attched with it.
>> I will let you know after ready! :)
>> 
>> Best,
>> Zhijiang
>> 
>> 
>> --
>> 发件人：Till Rohrmann 
>> 发送时间：2018年9月7日(星期五) 22:01
>> 收件人：Zhijiang(wangzhijiang999) 
>> 抄 送：dev 
>> 主 题：Re: [DISCUSS] Proposal of external shuffle service
>> 
>> The rough plan sounds good Zhijiang. I think we should continue with what
>> you've proposed: Open a JIRA issue and creating a design document which
>> outlines the required changes a little bit more in detail. Once this is
>> done, we should link the design document in the JIRA issue and post it here
>> for further discussion.
>> 
>> Cheers,
>> Till
>> 
>> On Wed, Aug 29, 2018 at 6:04 PM Zhijiang(wangzhijiang999) <
>> wangzhijiang...@aliyun.com> wrote:
>> 
>>> Glad to receive your positive feedbacks Till!
>>> 
>>> Actually our motivation is to support batch job well as you mentioned.
>>> 
>>> For output level, flink already has the Subpartition abstraction(writer),
>>> and currently there are PipelinedSubpartition(memory output) and
>>> SpillableSubpartition(one-sp-one-file output) implementations. We can
>>> extend this abstraction to realize other persistent outputs (e.g.
>>> sort-merge-file).
>>> 
>>> For transport level(shuffle service), the current SubpartitionView
>>> abstraction(reader) seems as the brige linked with the output level, then
>> 
>>> the view can understand and read the different output formats. The current
>>> NetworkEnvironment seems take the role of internal shuffle service in
>>> TaskManager and the transport server is realized by netty inside. This
>> 
>>> component can also be started in other external containers like NodeManager
>>> of yarn to take the role of external shuffle service. Further we can
>> 
>>> abstract to extend the shuffle service for transporting outputs by http or
>> 
>>> rdma instead of current netty.  This abstraction should provide the way for
>>> output registration in order to read the results correctly, similar with
>>> current SubpartitionView.
>>> 
>>> The above is still a rough idea. Next I plan to create a feature jira to
>>> cover the related changes if possible. It would be better if getting help
>>> from related committers to review the detail designs together.
>>> 
>>> Best,
>>> Zhijiang
>>> 
>>> --
>>> 发件人：Till Rohrmann 
>>> 发送时间：2018年8月29日(星期三) 17:36
>>> 收件人：dev ; Zhijiang(wangzhijiang999) <
>>> wangzhijiang...@aliyun.com>
>>> 主 题：Re: [DISCUSS] Proposal of external shuffle service
>>> 
>>> Thanks for starting this design discussion Zhijiang!
>>> 
>>> I really like the idea to introduce a ShuffleService abstraction which
>> 
>>> allows to have different implementations depending on the actual use case.
>> 
>>> Especially for batch jobs I can clearly see the benefits of persisting the
>>> results somewhere else.
>>> 
>>> Do you already know which interfaces we need to extend and where to
>>> introduce new abstractions?
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Mon, Aug 27, 2018 at 1:57 PM Zhijiang(wangzhijiang999)
>>>  wrote:
>>> Hi all!
>>> 
>> 
>>> The shuffle service is responsible for transporting upstream produced data
>>> to the downstream side. In flink, the NettyServer is used for network
>> 
>>> transport service and this component is started in the TaskManager 

[DISCUSS] Enhancing the functionality and productivity of Table API

[jincheng sun]

Hi all,

With the continuous efforts from the community, the Flink system has been
continuously improved, which has attracted more and more users. Flink SQL
is a canonical, widely used relational query language. However, there are
still some scenarios where Flink SQL failed to meet user needs in terms of
functionality and ease of use, such as:


   -

   In terms of functionality

Iteration, user-defined window, user-defined join, user-defined
GroupReduce, etc. Users cannot express them with SQL;

   -

   In terms of ease of use
   -

  Map - e.g. “dataStream.map(mapFun)”. Although “table.select(udf1(),
  udf2(), udf3())” can be used to accomplish the same function., with a
  map() function returning 100 columns, one has to define or call 100 UDFs
  when using SQL, which is quite involved.
  -

  FlatMap -  e.g. “dataStrem.flatmap(flatMapFun)”. Similarly, it can be
  implemented with “table.join(udtf).select()”. However, it is obvious that
  datastream is easier to use than SQL.


Due to the above two reasons, some users have to use the DataStream API or
the DataSet API. But when they do that, they lose the unification of batch
and streaming. They will also lose the sophisticated optimizations such as
codegen, aggregate join transpose  and multi-stage agg from Flink SQL.

We believe that enhancing the functionality and productivity is vital for
the successful adoption of Table API. To this end,  Table API still
requires more efforts from every contributor in the community. We see great
opportunity in improving our user’s experience from this work. Any feedback
is welcome.

Regards,

Jincheng

[jira] [Created] (FLINK-10745) Improve record writer and serializer stack

[zhijiang (JIRA)]

zhijiang created FLINK-10745:


 Summary: Improve record writer and serializer stack
 Key: FLINK-10745
 URL: https://issues.apache.org/jira/browse/FLINK-10745
 Project: Flink
  Issue Type: Improvement
  Components: Network
Affects Versions: 1.8.0
Reporter: zhijiang
Assignee: zhijiang


This is the umbrella issue for tracking the improvements of record writer and 
serialization stack.

 

There are mainly involved in three components:

1. {{ChannelSelector}} : Refactor the interface method and unify the 
implementations of stream and batch modes.

2. {{RecordSerializer}} : Redesign the process for serialization and copy only 
once for multiple selected channels.

2. {{RecordWriter}} : Redesign the process for the serialization improvement 
above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)