Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

[lining jing]

I think we can create the PR for the GC status later if we could find an
easy way to obtain it, before that the users could get GC logs from the
FLIP-103.

By the way, there is a similar topic 'FlameGraph In Job Vertex' in FLIP-75
in the early discussion stage
https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit#heading=h.6xcek9hyxzu0,
we move it into another FLIP to discuss later since FLIP-75 is heavy enough.

And I have created FLINK-15422
 to get more metrics for
JVM's memory.
Piotr Nowojski  于2020年2月18日周二 下午7:11写道：

> Hi,
>
> A quick question/comment about FLIP-102. Have you thought about adding GC
> stats? I’m not sure what’s easily do-able, but something that would allow
> user to see GC issues (long/frequent pauses, lots of CPU time spent in the
> GC) would be quite useful for analysing performance/stability issues,
> without a need of connecting profilers in a distributed environment?
>
> Piotrek
>
> > On 10 Feb 2020, at 10:58, Yadong Xie  wrote:
> >
> > Hi all
> > I have drafted the docs of top-level FLIPs for the individual changes
> > proposed in FLIP-75.
> > will update it to the cwiki page and start the voting stage soon if there
> > is no objection.
> >
> >   - FLIP-98: Better Back Pressure Detection
> >   <
> https://docs.google.com/document/d/1b4GadCze-36x5TPHz6ie4WI9fOUuxWoT_rWWgeg68oo/edit?usp=sharing
> >
> >   - FLIP-99: Make Max Exception Configurable
> >   <
> https://docs.google.com/document/d/1tsPpTEx5WqliOAUC924xzRxYOalUuB-GoznGPcxSzJo/edit?usp=sharing
> >
> >   - FLIP-100: Add Attempt Information
> >   <
> https://docs.google.com/document/d/1Ww7biOr6WMVfoYhtBTJftRqEm9FGo33AXgYibdXy47Y/edit?usp=sharing
> >
> >   - FLIP-101: Add Pending Slots Tab in Job Detail
> >   <
> https://docs.google.com/document/d/1ttn7zIn_Z237JOHdmhiei6aCwKdjTU53I07XxA61Fis/edit?usp=sharing
> >
> >   - FLIP-102: Add More Metrics to TaskManager
> >   <
> https://docs.google.com/document/d/18yHdsqUJ1FmNRm0hyeCm3nWvPFpvpJgTJ8BYNAa6Ul8/edit?usp=sharing
> >
> >   - FLIP-103: Better Taskmanager Log Display
> >   <
> https://docs.google.com/document/d/16eEdW2KeLxvABdoXahx4MMMisW4_P9mKiqUE0F4GO1c/edit?usp=sharing
> >
> >   - FLIP-104: Add More Metrics to Jobmanager
> >   <
> https://docs.google.com/document/d/1Fak632iOroOLZFADqwZWu2SS-LUQqCLHnm8Vs3XM5to/edit?usp=sharing
> >
> >   - FLIP-105: Better Jobmanager Log Display
> >   <
> https://docs.google.com/document/d/1ayXaZflelaymQuF3l6UOuGEg6zzbSGGewoDq-9SBOPY/edit?usp=sharing
> >
> >
> >
> > Yadong Xie  于2020年2月9日周日 下午7:24写道：
> >
> >> Hi Till
> >> I got your point, will create sub FLIPs and votings according to the
> >> FLIP-75 and previous discussion soon.
> >>
> >> Till Rohrmann  于2020年2月9日周日 下午5:27写道：
> >>
> >>> Hi Yadong,
> >>>
> >>> I think it would be fine to simply link to this discussion thread to
> keep
> >>> the discussion history. Maybe an easier way would be to create
> top-level
> >>> FLIPs for the individual changes proposed in FLIP-75. The reason I'm
> >>> proposing this is that it would be easier to vote on it and to
> implement
> >>> it
> >>> because the scope is smaller. But maybe I'm wrong here and others could
> >>> chime in to voice their opinion.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Fri, Feb 7, 2020 at 9:58 AM Yadong Xie  wrote:
> >>>
>  Hi Till
> 
>  FLIP-75 has been open since September, and the design doc has been
> >>> iterated
>  over 3 versions and more than 20 patches.
>  I had a try, but it is hard to split the design docs into sub FLIP and
> >>> keep
>  all the discussion history at the same time.
> 
>  Maybe it is better to start another discussion to talk about the
> >>> individual
>  sub FLIP voting? and make the next FLIP follow the new practice if
>  possible.
> 
>  Till Rohrmann  于2020年2月3日周一 下午6:28写道：
> 
> > I think there is no such description because we never did it before.
> I
>  just
> > figured that FLIP-75 could actually be a good candidate to start this
> > practice. We would need a community discussion first, though.
> >
> > Cheers,
> > Till
> >
> > On Mon, Feb 3, 2020 at 10:28 AM Yadong Xie 
> >>> wrote:
> >
> >> Hi Till
> >> I didn’t find how to create of sub flip at cwiki.apache.org
> >> do you mean to create 9 more FLIPS instead of FLIP-75?
> >>
> >> Till Rohrmann  于2020年1月30日周四 下午11:12写道：
> >>
> >>> Would it be easier if FLIP-75 would be the umbrella FLIP and we
> >>> would
> >> vote
> >>> on the individual improvements as sub FLIPs? Decreasing the scope
> > should
> >>> make things easier.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Thu, Jan 30, 2020 at 2:35 PM Robert Metzger <
> >>> rmetz...@apache.org>
> >>> wrote:
> >>>
>  Thanks a lot for this work! I believe the web UI is very
> >>> important,
> > in
>  particular to new users. I'm very 

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

[jincheng sun]

Thanks a lot for the release Chesnay!
And thanks to everyone who make this release possible!

Best,
Jincheng


Chesnay Schepler  于2020年2月19日周三 上午12:45写道：

> The Apache Flink community is very happy to announce the release of
> Apache Flink-shaded 10.0.
>
> The flink-shaded project contains a number of shaded dependencies for
> Apache Flink.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data
> streaming applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
>
> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
>
> Regards,
> Chesnay
>
>

[jira] [Created] (FLINK-16159) Add simple end-to-end test for Stateful Functions

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-16159:
---

 Summary: Add simple end-to-end test for Stateful Functions
 Key: FLINK-16159
 URL: https://issues.apache.org/jira/browse/FLINK-16159
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions, Test Infrastructure
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


Often in our review process of changes to the Stateful Functions project, we 
often end up wanting to do a simple sanity check by running the simple greeter 
example with docker-compose.

Ideally, this should really be an end-to-end verification program that:
* Starts a Kafka broker and a simple stateful function application that has a 
Kafka ingress and egress
* Uses the Kafka Java client to write some inputs to the broker, reads and 
verifies output

With testcontainers (https://www.testcontainers.org/) this might even be 
achievable as a Junit test, which runs only after the Maven packaging phase.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Improvements on FLIP Process

[jincheng sun]

Hi all,

Thanks for bring up the discussion @Hequn!

I agree with some concerns raised above, however, I would like to give my
+1 for the proposal and I would like to share my thoughts:

If I understand correctly, the proposal doesn’t encourage people to discuss
in the google doc, the first step of the proposal is to raise the
discussion on the mailing list.

It’s common sense to discuss on the mailing list even with a google doc.
This is also the current status and works well. Most people know that we
should focus the discussion on the mailing list especially for those about
architecture or something pretty important for discussing which is what we
want to left the history.

I believe the google doc brings more benefits for us than costs. The
problem is how we use it, not eliminate it. There are still some benefits
that we can get from it. For example, It is a good place to comment on the
document for some minor problems, e.g., typos or grammatical problems.
Correcting these problems could help us to achieve a high-quality document.
It is also unnecessarily to left history for these kinds of problems. If we
put all these comments into the mailing list. The mailing list would be
flooded. Meanwhile, it’s hard to comment on these problems on the mailing
list if the document is very long.

As for the FLIP process, it’s a good idea to make our wiki “immutable” so
that we can make the wiki management better, i.e., only editable by PMC or
committer(This can be discussed in another thread).

What do you think? Would be great if more people can share thoughts here!

Best,
Jincheng


Yuan Mei  于2020年2月19日周三 下午12:41写道：

> It is difficult to draw a clear cut between small and big issues. Hence I
> would prefer to stick to only one way for discussion.
>
> I would try to avoid Google Docs if having other ways mainly because of two
> reasons:
>
> 1. Google Docs are not always accessible to everyone.
>
> 2. Discussion on Google docs is difficult to track
> - new comments are notified through email
> - discussion history is hard to follow once a comment is resolved
> - limited spaces on the page to display e.t.c
>
>
> Best
>
> Yuan
>
> On Wed, Feb 19, 2020 at 11:52 AM Jingsong Li 
> wrote:
>
> > Hi all, thanks for launching this discussion.
> >
> > About eliminating Google Docs. I agree with Zhijiang, I share my concern
> > about it.
> >
> > If the FLIP Driver is a Flink newer or the FLIP is very big and
> > complicated. His/Her design maybe need change many many things, in this
> > situation, Google doc is good to be reviewed by community. If all
> > discussions are in ML, It's going to be very messy.
> >
> > So I think can keep this principle:
> > - Small issues can be discussed on Google doc.
> > - Big issues, or fundamental design issues, or API issues, are discussed
> in
> > ML.
> >
> > Best,
> > Jingsong Lee
> >
> > On Wed, Feb 19, 2020 at 1:22 AM Zhijiang  > .invalid>
> > wrote:
> >
> > > Thanks for launching this discussion and also agree with the opinions
> of
> > > Kostas, Timo and Aljoscha.
> > >
> > > The proposed reasons for eliminating google doc are very reasonable,
> > > especially the access limitation for some people in China.
> > >
> > > Besides that, another conservative option is to make google doc as an
> > > optional procedure, not a must procedure in practice, and
> > > the ML discussion is still the formal must procedure to follow firstly.
> > > And we can also kindly list these specific considerations/reasons
> > > for google doc concerns as said below in the guideline doc.
> > >
> > > To do so, we still retain this option for some people who prefer to
> > google
> > > doc or willing to provide it in some corner cases.
> > > Of course I am also happy to eliminate google doc completely.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:Kostas Kloudas 
> > > Send Time:2020 Feb. 18 (Tue.) 23:03
> > > To:dev 
> > > Subject:Re: [DISCUSS] Improvements on FLIP Process
> > >
> > > +1 to what Aljoscha and Timo are proposing.
> > >
> > > I would lean towards eliminating Google Docs altogether.
> > > I think they served a purpose when discussions were among 3 to 4
> > > people but with the current size of the community and the amount of
> > > participants per discussion they become difficult to follow.
> > >
> > > Best,
> > > Kostas
> > >
> > > On Tue, Feb 18, 2020 at 3:36 PM Timo Walther 
> wrote:
> > > >
> > > > +1 to what Aljoscha said.
> > > >
> > > > The past has shown that discussions in Google docs do not reach all
> > > > interested parties and the tracability of design decisions becomes
> > > > difficult. Google services are also partially inaccessible in certain
> > > > parts of world.
> > > >
> > > > We should actually do the opposite and not allow Google docs as FLIPs
> > > > anymore. Commenting should be disabled by default.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > >
> > > >
> > > > On 18.02.20

Re: [DISCUSS] Improvements on FLIP Process

[Yuan Mei]

It is difficult to draw a clear cut between small and big issues. Hence I
would prefer to stick to only one way for discussion.

I would try to avoid Google Docs if having other ways mainly because of two
reasons:

1. Google Docs are not always accessible to everyone.

2. Discussion on Google docs is difficult to track
- new comments are notified through email
- discussion history is hard to follow once a comment is resolved
- limited spaces on the page to display e.t.c


Best

Yuan

On Wed, Feb 19, 2020 at 11:52 AM Jingsong Li  wrote:

> Hi all, thanks for launching this discussion.
>
> About eliminating Google Docs. I agree with Zhijiang, I share my concern
> about it.
>
> If the FLIP Driver is a Flink newer or the FLIP is very big and
> complicated. His/Her design maybe need change many many things, in this
> situation, Google doc is good to be reviewed by community. If all
> discussions are in ML, It's going to be very messy.
>
> So I think can keep this principle:
> - Small issues can be discussed on Google doc.
> - Big issues, or fundamental design issues, or API issues, are discussed in
> ML.
>
> Best,
> Jingsong Lee
>
> On Wed, Feb 19, 2020 at 1:22 AM Zhijiang  .invalid>
> wrote:
>
> > Thanks for launching this discussion and also agree with the opinions of
> > Kostas, Timo and Aljoscha.
> >
> > The proposed reasons for eliminating google doc are very reasonable,
> > especially the access limitation for some people in China.
> >
> > Besides that, another conservative option is to make google doc as an
> > optional procedure, not a must procedure in practice, and
> > the ML discussion is still the formal must procedure to follow firstly.
> > And we can also kindly list these specific considerations/reasons
> > for google doc concerns as said below in the guideline doc.
> >
> > To do so, we still retain this option for some people who prefer to
> google
> > doc or willing to provide it in some corner cases.
> > Of course I am also happy to eliminate google doc completely.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Kostas Kloudas 
> > Send Time:2020 Feb. 18 (Tue.) 23:03
> > To:dev 
> > Subject:Re: [DISCUSS] Improvements on FLIP Process
> >
> > +1 to what Aljoscha and Timo are proposing.
> >
> > I would lean towards eliminating Google Docs altogether.
> > I think they served a purpose when discussions were among 3 to 4
> > people but with the current size of the community and the amount of
> > participants per discussion they become difficult to follow.
> >
> > Best,
> > Kostas
> >
> > On Tue, Feb 18, 2020 at 3:36 PM Timo Walther  wrote:
> > >
> > > +1 to what Aljoscha said.
> > >
> > > The past has shown that discussions in Google docs do not reach all
> > > interested parties and the tracability of design decisions becomes
> > > difficult. Google services are also partially inaccessible in certain
> > > parts of world.
> > >
> > > We should actually do the opposite and not allow Google docs as FLIPs
> > > anymore. Commenting should be disabled by default.
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > >
> > > On 18.02.20 15:20, Aljoscha Krettek wrote:
> > > > Hi,
> > > >
> > > > thanks for starting this discussion!
> > > >
> > > > However, I have a somewhat opposing opinion to this: we should
> disallow
> > > > using Google Docs for FLIPs and FLIP discussions and follow the
> already
> > > > established process more strictly.
> > > >
> > > > My reasons for this are:
> > > >   - discussions on the Google Doc are not reflected in Apache
> > > > infrastructure
> > > >   - discussions on Google Docs are non-linear and hard to follow
> > > >   - when discussions on Google Docs are resolved these discussions
> are
> > > > not visible/re-readable anymore (I know there's history, but meh)
> > > >   - if discussion is kept purely to the ML this is easily observable
> > for
> > > > any interested parties and it's there if somewhat want's to recheck
> the
> > > > discussion in the future
> > > >   - going from Google Doc to Wiki is an extra step that seems
> > > > unnecessary to me (but that's just personal opinion, I mean, I don't
> > > > have to do the extra work here...)
> > > >
> > > > Best,
> > > > Aljoscha
> > > >
> > > > On 18.02.20 09:02, Hequn Cheng wrote:
> > > >> Hi everyone,
> > > >>
> > > >> Currently, when we create a FLIP we follow the FLIP process in the
> > Flink
> > > >> Improvement Proposals wiki[1]. The process mainly includes the
> > following
> > > >> steps:
> > > >> 1. Create a FLIP wiki page.
> > > >> 2. Raise the discussion on the mailing list.
> > > >> 3. Once the proposal is finalized, call a vote to have the proposal
> > > >> adopted.
> > > >> There is also a discussion[2] on the FLIP process which may be
> helpful
> > > >> for
> > > >> you.
> > > >>
> > > >> As it is not allowed commented on the wiki, we usually have a google
> > doc
> > > >> for the discussion at step 2 and whenever there is a chan

Re: [DISCUSS] Improvements on FLIP Process

[Jingsong Li]

Hi all, thanks for launching this discussion.

About eliminating Google Docs. I agree with Zhijiang, I share my concern
about it.

If the FLIP Driver is a Flink newer or the FLIP is very big and
complicated. His/Her design maybe need change many many things, in this
situation, Google doc is good to be reviewed by community. If all
discussions are in ML, It's going to be very messy.

So I think can keep this principle:
- Small issues can be discussed on Google doc.
- Big issues, or fundamental design issues, or API issues, are discussed in
ML.

Best,
Jingsong Lee

On Wed, Feb 19, 2020 at 1:22 AM Zhijiang 
wrote:

> Thanks for launching this discussion and also agree with the opinions of
> Kostas, Timo and Aljoscha.
>
> The proposed reasons for eliminating google doc are very reasonable,
> especially the access limitation for some people in China.
>
> Besides that, another conservative option is to make google doc as an
> optional procedure, not a must procedure in practice, and
> the ML discussion is still the formal must procedure to follow firstly.
> And we can also kindly list these specific considerations/reasons
> for google doc concerns as said below in the guideline doc.
>
> To do so, we still retain this option for some people who prefer to google
> doc or willing to provide it in some corner cases.
> Of course I am also happy to eliminate google doc completely.
>
> Best,
> Zhijiang
>
>
> --
> From:Kostas Kloudas 
> Send Time:2020 Feb. 18 (Tue.) 23:03
> To:dev 
> Subject:Re: [DISCUSS] Improvements on FLIP Process
>
> +1 to what Aljoscha and Timo are proposing.
>
> I would lean towards eliminating Google Docs altogether.
> I think they served a purpose when discussions were among 3 to 4
> people but with the current size of the community and the amount of
> participants per discussion they become difficult to follow.
>
> Best,
> Kostas
>
> On Tue, Feb 18, 2020 at 3:36 PM Timo Walther  wrote:
> >
> > +1 to what Aljoscha said.
> >
> > The past has shown that discussions in Google docs do not reach all
> > interested parties and the tracability of design decisions becomes
> > difficult. Google services are also partially inaccessible in certain
> > parts of world.
> >
> > We should actually do the opposite and not allow Google docs as FLIPs
> > anymore. Commenting should be disabled by default.
> >
> > Regards,
> > Timo
> >
> >
> >
> > On 18.02.20 15:20, Aljoscha Krettek wrote:
> > > Hi,
> > >
> > > thanks for starting this discussion!
> > >
> > > However, I have a somewhat opposing opinion to this: we should disallow
> > > using Google Docs for FLIPs and FLIP discussions and follow the already
> > > established process more strictly.
> > >
> > > My reasons for this are:
> > >   - discussions on the Google Doc are not reflected in Apache
> > > infrastructure
> > >   - discussions on Google Docs are non-linear and hard to follow
> > >   - when discussions on Google Docs are resolved these discussions are
> > > not visible/re-readable anymore (I know there's history, but meh)
> > >   - if discussion is kept purely to the ML this is easily observable
> for
> > > any interested parties and it's there if somewhat want's to recheck the
> > > discussion in the future
> > >   - going from Google Doc to Wiki is an extra step that seems
> > > unnecessary to me (but that's just personal opinion, I mean, I don't
> > > have to do the extra work here...)
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 18.02.20 09:02, Hequn Cheng wrote:
> > >> Hi everyone,
> > >>
> > >> Currently, when we create a FLIP we follow the FLIP process in the
> Flink
> > >> Improvement Proposals wiki[1]. The process mainly includes the
> following
> > >> steps:
> > >> 1. Create a FLIP wiki page.
> > >> 2. Raise the discussion on the mailing list.
> > >> 3. Once the proposal is finalized, call a vote to have the proposal
> > >> adopted.
> > >> There is also a discussion[2] on the FLIP process which may be helpful
> > >> for
> > >> you.
> > >>
> > >> As it is not allowed commented on the wiki, we usually have a google
> doc
> > >> for the discussion at step 2 and whenever there is a change we need to
> > >> pick
> > >> it to the wiki page. This makes things somehow redundant. To solve
> > >> this, we
> > >> can rearrange the step a little bit and avoid the pick:
> > >> 1. Raise the discussion on the mailing list. The subject of the
> thread is
> > >> of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design
> doc
> > >> should follow the FLIP-Template strictly. (The [FLIP] tag is used to
> > >> inform
> > >> people that it is a FLIP discussion and more attention should be
> paid.)
> > >> 2. Create a FLIP wiki page once we reached an agreement on the
> > >> discussion.
> > >> We can simply copy the google doc into the FLIP wiki page.
> > >> 3. Once the proposal is finalized, call a vote to have the proposal
> > >> adopted. It should be noted that we should always vote on 

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Zhijiang]

Thanks for kicking off the next release and the introduction, @Stephan!

It's my pleasure to become the release manager and involve in other community 
works. I am working together with @Piotr for a long time,  so very happy to 
cooperate for the release manager work again. The previous release work was 
always well done, and I can learn a lot from these rich experiences.

+1 for the "feature freeze date" around end of April.
 Although we have the FF SF in the meantime, fortunately there are no long 
public holidays during this period in China.

Best,
Zhijiang


--
From:Stephan Ewen 
Send Time:2020 Feb. 19 (Wed.) 01:15
To:dev 
Cc:zhijiang ; pnowojski 
Subject:[DISCUSS] Kicking off the 1.11 release cycle

Hi all!

Now that the 1.10 release is out (congrats again, everyone!), I wanted to bring 
up some questions about organizing the next release cycle.

The most important questions at the beginning would be
  - Who would volunteer as Release Managers
  - When would be the release date.

For the release managers, Piotrek and Zhijiang have mentioned previously that 
they would be interested, so I am copying them here to chime in.
@Piotr and @Zhijiang could you confirm if you are interested in helping out 
with this?

About the release date: By our original "3 months release cycle" assumption, we 
should aim for a release **Mid May**, meaning a feature freeze no later than 
end of April.
That would be indeed a shorter release cycle than 1.10, and the assumption of a 
shorter testing period. But aiming for a shorter release cycle than 1.10 would 
actually be nice, in my opinion. 1.10 grew very big in the end, which caused 
also a very long testing period (plus Christmas and Chinese New Year are also 
partially to blame).

The exact feature freeze date is anyways a community discussion later, but what 
do you think about starting with an "anticipated feature freeze date" around 
end of April, so that committers, contributors, and users know roughly what to 
expect?

Best,
Stephan

[jira] [Created] (FLINK-16158) SqlClient showdown when executing update if not start cluster

[hailong wang (Jira)]

hailong wang created FLINK-16158:


 Summary: SqlClient showdown when executing update if not start 
cluster
 Key: FLINK-16158
 URL: https://issues.apache.org/jira/browse/FLINK-16158
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.10.0
Reporter: hailong wang
 Fix For: 1.11.0


If not start cluster, we execute update just as follow:
{code:java}
insert into sinks Select product from sources;
{code}
It will throw error:
{code:java}
Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
Unexpected exception. This is a bug. Please consider filing an issue.Exception 
in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected 
exception. This is a bug. Please consider filing an issue. at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)Caused by: 
java.lang.RuntimeException: Error running SQL job. at 
org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:606)
 at 
org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
 at org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:537) 
at org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:299) 
at java.util.Optional.ifPresent(Optional.java:159) at 
org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) at 
org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125) at 
org.apache.flink.table.client.SqlClient.start(SqlClient.java:104) at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)Caused by: 
java.util.concurrent.ExecutionException: 
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit 
JobGraph. at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at 
org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:603)
 ... 8 moreCaused by: org.apache.flink.runtime.client.JobSubmissionException: 
Failed to submit JobGraph. at 
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:359)
 at 
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
 at 
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
 at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
 at 
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:287)
 at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
 at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
 at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
 at 
org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
 at 
org.apache.flink.shaded.netty4.io.netty.util.inter

Re: Total recovery time estimation after checkpoint recovery

[Woods, Jessica Hui]

Hi Till,

No, I have not received any emails regarding my question. Could you please 
forward your response to me?

Thanks

From: Till Rohrmann 
Sent: Tuesday, February 18, 2020 4:43 PM
To: dev
Subject: Re: Total recovery time estimation after checkpoint recovery

Hi Jessica,

did you receive my previous email with the explanation?

Cheers,
Till

On Sat, Feb 15, 2020 at 11:45 PM Woods, Jessica Hui <
jessica.wo...@campus.tu-berlin.de> wrote:

> ??Hi,
>
> I am working with Apache Flink and am interested in knowing how one could
> estimate the total amount of time an application spends in recovery,
> including the input stream "catch-up" after checkpoint recovery. What I am
> specifically interested in is knowing the time needed for the recovery of
> the state + the catch-up phase (since the application's source tasks are
> reset to an earlier input position after recovery, this would be the data
> it processed before the failure and data that accumulated while the
> application was down).
>
> My question is, what important considerations should I take into account
> when estimating this time and which portions of the Apache Flink codebase
> would be most helpful?
>
> Thanks
>
>

[jira] [Created] (FLINK-16157) StateFun benchmark and performance

[Omid B (Jira)]

Omid B created FLINK-16157:
--

 Summary: StateFun benchmark and performance
 Key: FLINK-16157
 URL: https://issues.apache.org/jira/browse/FLINK-16157
 Project: Flink
  Issue Type: Wish
  Components: Stateful Functions
Reporter: Omid B


Hi,

Is there any project or attempt to see the performance of the StateFun 
functions. Throughput, the state change latency, etc ...

Thanks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Hotfixes on the master

[Till Rohrmann]

Thanks for raising this point Robert. I think it is important to remind the
community about the agreed practices once in a while.

In most of the cases I had the impression that the majority of the
community sticks to the agreed rules. W/o more detailed numbers (how many
of the hotfix commits are of category b) I think this discussion makes only
limited sense. Moreover, what is the underlying problem you would like to
solve here? Is it that too many commits have the hotfix tag in the commit
log? Is it that it's hard to figure out with which PR a hotfix commit has
been merged? Or did you observe a decline in Flink's quality because of too
many unreviewed changes? If we are discussing the latter case, then I think
it is very urgent. In the former cases I'm not entirely sure whether this
is an immediate problem because from what I have seen people include many
more clean up/hotfix commits in their PRs these days.

Concerning the proposed practice with the [minor] tag I'm a bit torn. The
benefit of not doing it like this is that it's easier to see which commits
need to be cherry-picked if a feature needs to be ported, for example. On
the other hand if the commits are not unrelated and the feature commits use
parts of the hotfix commits, then it makes the cherry-picking more tricky
and the commits should have the JIRA issue tag. But I would be fine with
trying it out.

Cheers,
Till

On Mon, Feb 17, 2020 at 3:56 PM Robert Metzger  wrote:

> Hi all,
>
> I would like to revive this very old thread on the topic of "unreviewed
> hotfixes on master" again.
> Out of the 35 commits listed on the latest commits page on GitHub, 18 have
> the tag "hotfix", on the next page it is 9, then 16, 17, ...
> In the last 140 commits, 42% were hotfixes.
>
> For the sake of this discussion, let's distinguish between two types of
> hotfixes:
> a) *reviewed hotfix commits*: They have been reviewed through a pull
> request, then committed to master.
> b) *unreviewed hotfix commits*: These have been pushed straight to master,
> without a review.
>
> It's quite difficult to find out whether a hotfix has been reviewed or not
> (because many hotfix commits are reviewed & pushed as part of a PR), but
> here are some recent examples of commits where I could not find evidence of
> a pull request:
>
> // these could probably be combined into on JIRA ticket, as they affect the
> same component + they touch dependencies
> 47a1725ae14a772ba8590ee97dffd7fdf5bc04b2 [hotfix][docs][conf] Log included
> packages / excluded classes
> a5894677d95336a67d5539584b9204bcdd14fac5 [hotfix][docs][conf] Setup logging
> for generator
> 325927064542c2d018f9da33660c1cdf57e0e382 [hotfix][docs][conf] Add query
> service port to port section
> 3c696a34145e838c046805b36553a50ec9bfbda0 [hotfix][docs][conf] Add query
> service port to port section
>
> // dependency change
> 736ebc0b40abab88902ada3f564777c3ade03001 [hotfix][build] Remove various
> unused test dependencies
>
> // more than a regeneration / typo / compile error change
> 30b5f6173e688ea20b82226db6923db19dec29a5 [hotfix][tests] Adjust
> FileUtilsTest.testDeleteSymbolicLinkDirectory() to handle unsupported
> situations in Windows
> fc59aa4ecc2a7170bfda14ffadf0a30aa2b793bf [FLINK-16065][core] Unignore
> FileUtilsTest.testDeleteDirectoryConcurrently()
>
> // dependency changes
> fe7145787a7f36b21aad748ffea4ee8ab03c02b7 [hotfix][build] Remove unused
> akka-testkit dependencies
> dd34b050e8e7bd4b03ad0870a432b1631e1c0e9d [hotfix][build] Remove unused
> shaded-asm7 dependencies
>
> // dependency changes
> 244d2db78307cd7dff1c60a664046adb6fe5c405 [hotfix][web][build] Cleanup
> dependencies
>
> In my opinion, everything that is not a typo, a compile error (breaking the
> master) or something generated (like parts of the docs) should go through a
> quick pull request.
> Why? I don't think many people review changes in the commit log in a way
> they review pull request changes.
>
> In addition to that, I propose to prefix hotfixes that have been added as
> part of a ticket with that ticket number.
> So instead of "[hotfix] Harden kubernetes test", we do
> "[FLINK-13978][minor]
> Harden kubernetes test".
> Why? For people checking the commit history, it is much easier to see if a
> hotfix has been reviewed as part of a JIRA ticket review, or whether it is
> a "hotpush" hotfix.
>
> For changes that are too small for a JIRA ticket, but need a review, I
> propose to use the "[minor]" tag. A good example of such a change is this:
>
> https://github.com/apache/flink/commit/0dc4e767c9c48ac58430a59d05185f2b071f53f5
>
> My tagging minor changes accordingly in the pull requests, it is also
> easier for fellow committers to quickly check them.
>
> Summary:
> [FLINK-]: regular, reviewed change
> [FLINK-][minor]: minor, unrelated changes reviewed with a regular
> ticket
> [minor]: minor, reviewed change
> [hotfix]: unreviewed change that fixes a typo, compile error or something
> generated
>
>
> What's your opinion on this?

[jira] [Created] (FLINK-16156) TableAggregateITCase.testGroupByFlatAggregate failed on Travis

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-16156:
-

 Summary: TableAggregateITCase.testGroupByFlatAggregate failed on 
Travis
 Key: FLINK-16156
 URL: https://issues.apache.org/jira/browse/FLINK-16156
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.1, 1.11.0


The {{TableAggregateITCase.testGroupByFlatAggregate}} failed because the order 
of the result did not match:

{code}
11:50:11.239 [ERROR] Failures: 
11:50:11.239 [ERROR]   FunctionITCase.testDynamicTableFunction:611 
Expected: <[Test is a string, 42, null]>
 but: was <[42, null, Test is a string]>
11:50:11.239 [ERROR] Errors: 
11:50:11.239 [ERROR]   TableAggregateITCase.testGroupByFlatAggregate:59 » 
JobExecution Job execution ...
11:50:11.239 [INFO] 
{code}

https://api.travis-ci.com/v3/job/288392903/log.txt



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16154) Translate "Operator/Join" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16154:
---

 Summary: Translate "Operator/Join" into Chinese
 Key: FLINK-16154
 URL: https://issues.apache.org/jira/browse/FLINK-16154
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is located at _"docs/dev/stream/operators/joining.zh.md"_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16155) Translate "Operator/Process Function" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16155:
---

 Summary: Translate "Operator/Process Function" into Chinese
 Key: FLINK-16155
 URL: https://issues.apache.org/jira/browse/FLINK-16155
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is located at _"docs/dev/stream/operators/process_function.zh.md"_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16153) Translate "Operator/windows" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16153:
---

 Summary: Translate "Operator/windows" into Chinese
 Key: FLINK-16153
 URL: https://issues.apache.org/jira/browse/FLINK-16153
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is located at _"docs/dev/stream/operators/windows.zh.md"_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16152) Translate "Operator/index" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16152:
---

 Summary: Translate "Operator/index" into Chinese
 Key: FLINK-16152
 URL: https://issues.apache.org/jira/browse/FLINK-16152
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is located at _docs/dev/stream/operators/index.zh.md_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Improvements on FLIP Process

[Zhijiang]

Thanks for launching this discussion and also agree with the opinions of 
Kostas, Timo and Aljoscha.

The proposed reasons for eliminating google doc are very reasonable, especially 
the access limitation for some people in China.

Besides that, another conservative option is to make google doc as an optional 
procedure, not a must procedure in practice, and
the ML discussion is still the formal must procedure to follow firstly. And we 
can also kindly list these specific considerations/reasons
for google doc concerns as said below in the guideline doc.

To do so, we still retain this option for some people who prefer to google doc 
or willing to provide it in some corner cases.
Of course I am also happy to eliminate google doc completely.

Best,
Zhijiang


--
From:Kostas Kloudas 
Send Time:2020 Feb. 18 (Tue.) 23:03
To:dev 
Subject:Re: [DISCUSS] Improvements on FLIP Process

+1 to what Aljoscha and Timo are proposing.

I would lean towards eliminating Google Docs altogether.
I think they served a purpose when discussions were among 3 to 4
people but with the current size of the community and the amount of
participants per discussion they become difficult to follow.

Best,
Kostas

On Tue, Feb 18, 2020 at 3:36 PM Timo Walther  wrote:
>
> +1 to what Aljoscha said.
>
> The past has shown that discussions in Google docs do not reach all
> interested parties and the tracability of design decisions becomes
> difficult. Google services are also partially inaccessible in certain
> parts of world.
>
> We should actually do the opposite and not allow Google docs as FLIPs
> anymore. Commenting should be disabled by default.
>
> Regards,
> Timo
>
>
>
> On 18.02.20 15:20, Aljoscha Krettek wrote:
> > Hi,
> >
> > thanks for starting this discussion!
> >
> > However, I have a somewhat opposing opinion to this: we should disallow
> > using Google Docs for FLIPs and FLIP discussions and follow the already
> > established process more strictly.
> >
> > My reasons for this are:
> >   - discussions on the Google Doc are not reflected in Apache
> > infrastructure
> >   - discussions on Google Docs are non-linear and hard to follow
> >   - when discussions on Google Docs are resolved these discussions are
> > not visible/re-readable anymore (I know there's history, but meh)
> >   - if discussion is kept purely to the ML this is easily observable for
> > any interested parties and it's there if somewhat want's to recheck the
> > discussion in the future
> >   - going from Google Doc to Wiki is an extra step that seems
> > unnecessary to me (but that's just personal opinion, I mean, I don't
> > have to do the extra work here...)
> >
> > Best,
> > Aljoscha
> >
> > On 18.02.20 09:02, Hequn Cheng wrote:
> >> Hi everyone,
> >>
> >> Currently, when we create a FLIP we follow the FLIP process in the Flink
> >> Improvement Proposals wiki[1]. The process mainly includes the following
> >> steps:
> >> 1. Create a FLIP wiki page.
> >> 2. Raise the discussion on the mailing list.
> >> 3. Once the proposal is finalized, call a vote to have the proposal
> >> adopted.
> >> There is also a discussion[2] on the FLIP process which may be helpful
> >> for
> >> you.
> >>
> >> As it is not allowed commented on the wiki, we usually have a google doc
> >> for the discussion at step 2 and whenever there is a change we need to
> >> pick
> >> it to the wiki page. This makes things somehow redundant. To solve
> >> this, we
> >> can rearrange the step a little bit and avoid the pick:
> >> 1. Raise the discussion on the mailing list. The subject of the thread is
> >> of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
> >> should follow the FLIP-Template strictly. (The [FLIP] tag is used to
> >> inform
> >> people that it is a FLIP discussion and more attention should be paid.)
> >> 2. Create a FLIP wiki page once we reached an agreement on the
> >> discussion.
> >> We can simply copy the google doc into the FLIP wiki page.
> >> 3. Once the proposal is finalized, call a vote to have the proposal
> >> adopted. It should be noted that we should always vote on a FLIP wiki
> >> page
> >> instead of a google doc. The wiki page is the final version of the google
> >> doc.
> >>
> >> This can bring some benefits:
> >> 1. Make the discussion more effective as we force people to write and
> >> discuss on a google doc that follows the FLIP template which
> >> includes necessary information such as Motivation, Interfaces, Proposed
> >> changes, etc.
> >> 2. Avoid redundant pick from google doc to Flink wiki page. Once we
> >> reached
> >> an agreement on the discussion, we can simply copy the google doc into
> >> the
> >> FLIP wiki page.
> >> 3. As adopted FLIP should mostly be "immutable", we can even make the
> >> wiki
> >> page PMC or committer editable since it just needs a simple copy from the
> >> google doc.
> >>
> >> Looking forward to your feedback!
> >>
> >> Best,
> >> Hequn
> >

[jira] [Created] (FLINK-16151) Translate "Event-time/Pre-defined Timestamp Extractors / Watermark Emitters" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16151:
---

 Summary: Translate "Event-time/Pre-defined Timestamp Extractors / 
Watermark Emitters" into Chinese
 Key: FLINK-16151
 URL: https://issues.apache.org/jira/browse/FLINK-16151
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is _docs/dev/event_timestamp_extractors.zh.md ._ 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Kicking off the 1.11 release cycle

[Stephan Ewen]

Hi all!

Now that the 1.10 release is out (congrats again, everyone!), I wanted to
bring up some questions about organizing the next release cycle.

The most important questions at the beginning would be
  - Who would volunteer as Release Managers
  - When would be the release date.

For the *release managers*, Piotrek and Zhijiang have mentioned previously
that they would be interested, so I am copying them here to chime in.
@Piotr and @Zhijiang could you confirm if you are interested in helping out
with this?

*About the release date*: By our original "3 months release cycle"
assumption, we should aim for a release ***Mid May***, meaning a feature
freeze no later than end of April.
That would be indeed a shorter release cycle than 1.10, and the assumption
of a shorter testing period. But aiming for a shorter release cycle than
1.10 would actually be nice, in my opinion. 1.10 grew very big in the end,
which caused also a very long testing period (plus Christmas and Chinese
New Year are also partially to blame).

The exact feature freeze date is anyways a community discussion later, but
what do you think about starting with an *"anticipated feature freeze date"*
around end of April, so that committers, contributors, and users know
roughly what to expect?

Best,
Stephan

[jira] [Created] (FLINK-16150) Eclipse improvements for archetypes break functionality in vscode-java

[Tristan Tunderman (Jira)]

Tristan Tunderman created FLINK-16150:
-

 Summary: Eclipse improvements for archetypes break functionality 
in vscode-java
 Key: FLINK-16150
 URL: https://issues.apache.org/jira/browse/FLINK-16150
 Project: Flink
  Issue Type: Bug
  Components: Quickstarts
Affects Versions: 1.10.0, 1.8.3
Reporter: Tristan Tunderman


The Maven archetypes (quickstarts) have some Eclipse oriented improvements in 
`pom.xml`. These are present to improve the "out-of-the-box experience in 
Eclipse by resolving some warnings" (direct citation from the file).

However, part of these improvements degrade the functionality in vscode in 
combination with the vscode-java extension. This extension provides Java 
language support in vscode (repository can be found here 
[https://github.com/redhat-developer/vscode-java] ).

The following piece of the XML
{code:java}


org.apache.maven.plugins
maven-compiler-plugin
[3.1,)

testCompile
compile






{code}
causes all of the autocomplete, code inspections, import fixes etc. provided by 
the vscode-java extension to be supressed.

I initially created an issue at the vscode-java repo, because I thought the 
issue was in the extension. However, they pointed out that the above piece of 
code causes the supressions (as mentioned here: 
[https://github.com/redhat-developer/vscode-java/issues/1241|https://github.com/redhat-developer/vscode-java/issues/1241)]
 ).

In the docs the preference for IntelliJ or Eclipse as an IDE is given, but only 
when contributing to the development of Flink itself, not for the applications 
using it. Long story short: is it either possible for the above piece of code 
to be removed (probably not), or can a warning be placed near it, or in the 
docs themselves, to prevent other people from stumbling across the same issue 
when they are using vscode?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[ANNOUNCE] Apache Flink-shaded 10.0 released

[Chesnay Schepler]

The Apache Flink community is very happy to announce the release of 
Apache Flink-shaded 10.0.


The flink-shaded project contains a number of shaded dependencies for 
Apache Flink.


Apache Flink® is an open-source stream processing framework for 
distributed, high-performing, always-available, and accurate data 
streaming applications.


The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746

We would like to thank all contributors of the Apache Flink community 
who made this release possible!


Regards,
Chesnay

[jira] [Created] (FLINK-16149) Set configurations using StreamExecutionEnvironment#getConfiguration

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-16149:


 Summary: Set configurations using 
StreamExecutionEnvironment#getConfiguration
 Key: FLINK-16149
 URL: https://issues.apache.org/jira/browse/FLINK-16149
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
 Fix For: statefun-1.1


FLIP-54 introduced utilities to set all configurations via flink-conf.

 

We should update StatefulFunctions to use this functionality so that statefun 
has consistent configurations on the dispatcher and TM's. We can also drop 
certain configurations, such as checkpointing, which are now covered by default 
flink configs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Total recovery time estimation after checkpoint recovery

[Till Rohrmann]

Hi Jessica,

did you receive my previous email with the explanation?

Cheers,
Till

On Sat, Feb 15, 2020 at 11:45 PM Woods, Jessica Hui <
jessica.wo...@campus.tu-berlin.de> wrote:

> ??Hi,
>
> I am working with Apache Flink and am interested in knowing how one could
> estimate the total amount of time an application spends in recovery,
> including the input stream "catch-up" after checkpoint recovery. What I am
> specifically interested in is knowing the time needed for the recovery of
> the state + the catch-up phase (since the application's source tasks are
> reset to an earlier input position after recovery, this would be the data
> it processed before the failure and data that accumulated while the
> application was down).
>
> My question is, what important considerations should I take into account
> when estimating this time and which portions of the Apache Flink codebase
> would be most helpful?
>
> Thanks
>
>

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[Till Rohrmann]

Thanks for updating the 1.9.2 release wrt Flink's Python API Jincheng!

Cheers,
Till

On Thu, Feb 13, 2020 at 12:25 PM Hequn Cheng  wrote:

> Thanks a lot for the release, Jincheng!
> Also thanks to everyone that make this release possible!
>
> Best,
> Hequn
>
> On Thu, Feb 13, 2020 at 2:18 PM Dian Fu  wrote:
>
> > Thanks for the great work, Jincheng.
> >
> > Regards,
> > Dian
> >
> > 在 2020年2月13日，下午1:32，jincheng sun  写道：
> >
> > Hi everyone,
> >
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for
> the
> > Apache Flink Python API 1.9 series.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> >
> > The release is available for download at:
> >
> > https://pypi.org/project/apache-flink/1.9.2/#files
> >
> > Or installed using pip command:
> >
> > pip install apache-flink==1.9.2
> >
> > We would like to thank all contributors of the Apache Flink community who
> > helped to verify this release and made this release possible!
> >
> > Best,
> > Jincheng
> >
> >
> >
>

Re: [DISCUSS] Improvements on FLIP Process

[Kostas Kloudas]

+1 to what Aljoscha and Timo are proposing.

I would lean towards eliminating Google Docs altogether.
I think they served a purpose when discussions were among 3 to 4
people but with the current size of the community and the amount of
participants per discussion they become difficult to follow.

Best,
Kostas

On Tue, Feb 18, 2020 at 3:36 PM Timo Walther  wrote:
>
> +1 to what Aljoscha said.
>
> The past has shown that discussions in Google docs do not reach all
> interested parties and the tracability of design decisions becomes
> difficult. Google services are also partially inaccessible in certain
> parts of world.
>
> We should actually do the opposite and not allow Google docs as FLIPs
> anymore. Commenting should be disabled by default.
>
> Regards,
> Timo
>
>
>
> On 18.02.20 15:20, Aljoscha Krettek wrote:
> > Hi,
> >
> > thanks for starting this discussion!
> >
> > However, I have a somewhat opposing opinion to this: we should disallow
> > using Google Docs for FLIPs and FLIP discussions and follow the already
> > established process more strictly.
> >
> > My reasons for this are:
> >   - discussions on the Google Doc are not reflected in Apache
> > infrastructure
> >   - discussions on Google Docs are non-linear and hard to follow
> >   - when discussions on Google Docs are resolved these discussions are
> > not visible/re-readable anymore (I know there's history, but meh)
> >   - if discussion is kept purely to the ML this is easily observable for
> > any interested parties and it's there if somewhat want's to recheck the
> > discussion in the future
> >   - going from Google Doc to Wiki is an extra step that seems
> > unnecessary to me (but that's just personal opinion, I mean, I don't
> > have to do the extra work here...)
> >
> > Best,
> > Aljoscha
> >
> > On 18.02.20 09:02, Hequn Cheng wrote:
> >> Hi everyone,
> >>
> >> Currently, when we create a FLIP we follow the FLIP process in the Flink
> >> Improvement Proposals wiki[1]. The process mainly includes the following
> >> steps:
> >> 1. Create a FLIP wiki page.
> >> 2. Raise the discussion on the mailing list.
> >> 3. Once the proposal is finalized, call a vote to have the proposal
> >> adopted.
> >> There is also a discussion[2] on the FLIP process which may be helpful
> >> for
> >> you.
> >>
> >> As it is not allowed commented on the wiki, we usually have a google doc
> >> for the discussion at step 2 and whenever there is a change we need to
> >> pick
> >> it to the wiki page. This makes things somehow redundant. To solve
> >> this, we
> >> can rearrange the step a little bit and avoid the pick:
> >> 1. Raise the discussion on the mailing list. The subject of the thread is
> >> of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
> >> should follow the FLIP-Template strictly. (The [FLIP] tag is used to
> >> inform
> >> people that it is a FLIP discussion and more attention should be paid.)
> >> 2. Create a FLIP wiki page once we reached an agreement on the
> >> discussion.
> >> We can simply copy the google doc into the FLIP wiki page.
> >> 3. Once the proposal is finalized, call a vote to have the proposal
> >> adopted. It should be noted that we should always vote on a FLIP wiki
> >> page
> >> instead of a google doc. The wiki page is the final version of the google
> >> doc.
> >>
> >> This can bring some benefits:
> >> 1. Make the discussion more effective as we force people to write and
> >> discuss on a google doc that follows the FLIP template which
> >> includes necessary information such as Motivation, Interfaces, Proposed
> >> changes, etc.
> >> 2. Avoid redundant pick from google doc to Flink wiki page. Once we
> >> reached
> >> an agreement on the discussion, we can simply copy the google doc into
> >> the
> >> FLIP wiki page.
> >> 3. As adopted FLIP should mostly be "immutable", we can even make the
> >> wiki
> >> page PMC or committer editable since it just needs a simple copy from the
> >> google doc.
> >>
> >> Looking forward to your feedback!
> >>
> >> Best,
> >> Hequn
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >>
> >> [2]
> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-META-FLIP-Sticking-or-not-to-a-strict-FLIP-voting-process-td29978.html#a29988
> >>
> >>
>

Re: [DISCUSS] Improvements on FLIP Process

[Timo Walther]

+1 to what Aljoscha said.

The past has shown that discussions in Google docs do not reach all 
interested parties and the tracability of design decisions becomes 
difficult. Google services are also partially inaccessible in certain 
parts of world.


We should actually do the opposite and not allow Google docs as FLIPs 
anymore. Commenting should be disabled by default.


Regards,
Timo



On 18.02.20 15:20, Aljoscha Krettek wrote:

Hi,

thanks for starting this discussion!

However, I have a somewhat opposing opinion to this: we should disallow 
using Google Docs for FLIPs and FLIP discussions and follow the already 
established process more strictly.


My reasons for this are:
  - discussions on the Google Doc are not reflected in Apache 
infrastructure

  - discussions on Google Docs are non-linear and hard to follow
  - when discussions on Google Docs are resolved these discussions are 
not visible/re-readable anymore (I know there's history, but meh)
  - if discussion is kept purely to the ML this is easily observable for 
any interested parties and it's there if somewhat want's to recheck the 
discussion in the future
  - going from Google Doc to Wiki is an extra step that seems 
unnecessary to me (but that's just personal opinion, I mean, I don't 
have to do the extra work here...)


Best,
Aljoscha

On 18.02.20 09:02, Hequn Cheng wrote:

Hi everyone,

Currently, when we create a FLIP we follow the FLIP process in the Flink
Improvement Proposals wiki[1]. The process mainly includes the following
steps:
1. Create a FLIP wiki page.
2. Raise the discussion on the mailing list.
3. Once the proposal is finalized, call a vote to have the proposal 
adopted.
There is also a discussion[2] on the FLIP process which may be helpful 
for

you.

As it is not allowed commented on the wiki, we usually have a google doc
for the discussion at step 2 and whenever there is a change we need to 
pick
it to the wiki page. This makes things somehow redundant. To solve 
this, we

can rearrange the step a little bit and avoid the pick:
1. Raise the discussion on the mailing list. The subject of the thread is
of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
should follow the FLIP-Template strictly. (The [FLIP] tag is used to 
inform

people that it is a FLIP discussion and more attention should be paid.)
2. Create a FLIP wiki page once we reached an agreement on the 
discussion.

We can simply copy the google doc into the FLIP wiki page.
3. Once the proposal is finalized, call a vote to have the proposal
adopted. It should be noted that we should always vote on a FLIP wiki 
page

instead of a google doc. The wiki page is the final version of the google
doc.

This can bring some benefits:
1. Make the discussion more effective as we force people to write and
discuss on a google doc that follows the FLIP template which
includes necessary information such as Motivation, Interfaces, Proposed
changes, etc.
2. Avoid redundant pick from google doc to Flink wiki page. Once we 
reached
an agreement on the discussion, we can simply copy the google doc into 
the

FLIP wiki page.
3. As adopted FLIP should mostly be "immutable", we can even make the 
wiki

page PMC or committer editable since it just needs a simple copy from the
google doc.

Looking forward to your feedback!

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals 


[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-META-FLIP-Sticking-or-not-to-a-strict-FLIP-voting-process-td29978.html#a29988 

Re: [DISCUSS] Improvements on FLIP Process

[Aljoscha Krettek]

Hi,

thanks for starting this discussion!

However, I have a somewhat opposing opinion to this: we should disallow 
using Google Docs for FLIPs and FLIP discussions and follow the already 
established process more strictly.


My reasons for this are:
 - discussions on the Google Doc are not reflected in Apache infrastructure
 - discussions on Google Docs are non-linear and hard to follow
 - when discussions on Google Docs are resolved these discussions are 
not visible/re-readable anymore (I know there's history, but meh)
 - if discussion is kept purely to the ML this is easily observable for 
any interested parties and it's there if somewhat want's to recheck the 
discussion in the future
 - going from Google Doc to Wiki is an extra step that seems 
unnecessary to me (but that's just personal opinion, I mean, I don't 
have to do the extra work here...)


Best,
Aljoscha

On 18.02.20 09:02, Hequn Cheng wrote:

Hi everyone,

Currently, when we create a FLIP we follow the FLIP process in the Flink
Improvement Proposals wiki[1]. The process mainly includes the following
steps:
1. Create a FLIP wiki page.
2. Raise the discussion on the mailing list.
3. Once the proposal is finalized, call a vote to have the proposal adopted.
There is also a discussion[2] on the FLIP process which may be helpful for
you.

As it is not allowed commented on the wiki, we usually have a google doc
for the discussion at step 2 and whenever there is a change we need to pick
it to the wiki page. This makes things somehow redundant. To solve this, we
can rearrange the step a little bit and avoid the pick:
1. Raise the discussion on the mailing list. The subject of the thread is
of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
should follow the FLIP-Template strictly. (The [FLIP] tag is used to inform
people that it is a FLIP discussion and more attention should be paid.)
2. Create a FLIP wiki page once we reached an agreement on the discussion.
We can simply copy the google doc into the FLIP wiki page.
3. Once the proposal is finalized, call a vote to have the proposal
adopted. It should be noted that we should always vote on a FLIP wiki page
instead of a google doc. The wiki page is the final version of the google
doc.

This can bring some benefits:
1. Make the discussion more effective as we force people to write and
discuss on a google doc that follows the FLIP template which
includes necessary information such as Motivation, Interfaces, Proposed
changes, etc.
2. Avoid redundant pick from google doc to Flink wiki page. Once we reached
an agreement on the discussion, we can simply copy the google doc into the
FLIP wiki page.
3. As adopted FLIP should mostly be "immutable", we can even make the wiki
page PMC or committer editable since it just needs a simple copy from the
google doc.

Looking forward to your feedback!

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-META-FLIP-Sticking-or-not-to-a-strict-FLIP-voting-process-td29978.html#a29988

[jira] [Created] (FLINK-16148) Update Operations Playground to Flink 1.10.0

[Oleg Bonar (Jira)]

Oleg Bonar created FLINK-16148:
--

 Summary: Update Operations Playground to Flink 1.10.0
 Key: FLINK-16148
 URL: https://issues.apache.org/jira/browse/FLINK-16148
 Project: Flink
  Issue Type: Task
  Components: Documentation
Affects Versions: 1.10.0
Reporter: Oleg Bonar


Update the operations playground to Flink 1.10.0
 This includes:
 * Updating the flink-playgrounds repository
 * Updating the "Getting Started/Docker Playgrounds" section in the 
documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16147) Add watermarkExprOutputType in WatermarkSpec#toString method

[hailong wang (Jira)]

hailong wang created FLINK-16147:


 Summary: Add watermarkExprOutputType in WatermarkSpec#toString 
method
 Key: FLINK-16147
 URL: https://issues.apache.org/jira/browse/FLINK-16147
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Common
Affects Versions: 1.10.0
Reporter: hailong wang
 Fix For: 1.11.0


WatermarkSpec#toString only include rowtimeAttribute and 
watermarkExpressionString.

Maybe we should inlcude watermarkExprOutputType too, just like:
{code:java}
@Override
public String toString() {
   return "rowtime: '" + rowtimeAttribute + '\'' +
  ", watermark: '" + watermarkExpressionString + '\'' +
  ", outputType: '" + watermarkExprOutputType + '\'';
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16146) Improve end-to-end usability of Flink Table API & SQL

[Jark Wu (Jira)]

Jark Wu created FLINK-16146:
---

 Summary: Improve end-to-end usability of Flink Table API & SQL
 Key: FLINK-16146
 URL: https://issues.apache.org/jira/browse/FLINK-16146
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Common, Table SQL / API, Table SQL / Client
Reporter: Jark Wu


This is an umbrella issue to track the usability problems in the end-to-end 
experience of Flink Table API & SQL. This may include API improvements or 
connectors improvements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

checkpoint total restart time

[Woods, Jessica Hui]

Hi,

I am working with Apache Flink and am interested in knowing how one could 
estimate the total amount of time an application spends in recovery, including 
the input stream "catch-up" after checkpoint recovery. What I am specifically 
interested in is knowing the time needed for the recovery of the state + the 
catch-up phase (since the application's source tasks are reset to an earlier 
input position after recovery, this would be the data it processed before the 
failure and data that accumulated while the application was down).

My question is, what important considerations should I take into account when 
estimating this time and which portions of the Apache Flink codebase would be 
most helpful?

Thanks?

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Chesnay Schepler]

Since one of the reasons for dropping ES2 was that it blocks some 
critical updates for the ES5 connector I'd prefer to keep ES5 around for 
1.11, and revisit this discussion for 1.12 .


On 18/02/2020 13:03, Aljoscha Krettek wrote:
Wouldn't removing the ES 2.x connector be enough because we can then 
update the ES 5.x connector? It seems there are some users that still 
want to use that one.


Best,
Aljoscha

On 18.02.20 10:42, Robert Metzger wrote:

The ES5 connector is causing some problems on the CI system. It would be
nice if we could make a decision here soon. I don't want to invest time
into fixing it, if we are going to remove it.

I'm still in favor of removing it. If we see that there's demand for the
5.x connector after the 1.11 release, somebody can take the source and
contribute it to Apache Bahir or a GitHub account and then posts it to
flink-packages.org.


On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz 


wrote:


Sorry for late reply,

@all I think there is a general consensus that we want to drop ES 2.x
support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
track it.


@Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
Client. ES promises to maintain compatibility of it with any newer 
minor

version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
etc.

ES provides also a low level rest client which does not include any
direct es dependencies and can work with any version of ES. It does not
provide any marshalling unmarshalling or higher level features as
Chesnay said.

Correct me if I am wrong @Itamar but your HTTP client is a simplified
version of the ES's high level rest client with a subset of its
features. I think it will still have the same problems as ES's High
Level Rest Client's because ES does not guarantee that newer message
formats will be compatible with older versions of ES or that message
formats are compatible across major versions at all.


@Stephan @Danny As for the 5.x connector. Any ideas how can we get
user's feedback about it? I cross posted on the user mailing list with
no luck so far. Personally I would be in favor of dropping the
connector. Worst case scenario users still have the possibility of
building the connector themselves from source with just bumping the
flink's versions. As far as I can tell there were no changes to the 
code

base for quite some time.

Best,

Dawid

On 11/02/2020 10:46, Chesnay Schepler wrote:

I suppose the downside in an HTTP ES sink is that you don't get _any_
form of high-level API from ES, and we'd have to manually build an
HTTP request that matches the ES format. Of course you also lose any
client-side verification that the clients did, if there is any (but I
guess the API itself prevented certain errors).

On 11/02/2020 09:32, Stephan Ewen wrote:
+1 to drop ES 2.x - unsure about 5.x (makes sense to get more user 
input

for that one).

@Itamar - if you would be interested in contributing a "universal" or
"cross version" ES connector, that could be very interesting. Do you
know
if there are known performance issues or feature restrictions with 
that

approach?
@dawid what do you think about that?


On Tue, Feb 11, 2020 at 6:28 AM Danny Chan 

wrote:


5.x seems to have a lot of users, is the 6.x completely 
compatible with

5.x ~

Best,
Danny Chan
在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
，写道：

Hi all,

As described in this

https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the 
box on

some systems and requires a version bump. This also happens for our
e2e.
We cannot bump the version in es 5.x connector, because 5.x 
connector

shares a common class with 2.x that uses an API that was replaced
in 5.2.

Both versions are already long eol:

https://www.elastic.co/support/eol


I suggest to drop both connectors 5.x and 2.x. If it is too much to
drop
both of them, I would strongly suggest dropping at least 2.x 
connector

and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Aljoscha Krettek]

Wouldn't removing the ES 2.x connector be enough because we can then 
update the ES 5.x connector? It seems there are some users that still 
want to use that one.


Best,
Aljoscha

On 18.02.20 10:42, Robert Metzger wrote:

The ES5 connector is causing some problems on the CI system. It would be
nice if we could make a decision here soon. I don't want to invest time
into fixing it, if we are going to remove it.

I'm still in favor of removing it. If we see that there's demand for the
5.x connector after the 1.11 release, somebody can take the source and
contribute it to Apache Bahir or a GitHub account and then posts it to
flink-packages.org.


On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz 
wrote:


Sorry for late reply,

@all I think there is a general consensus that we want to drop ES 2.x
support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
track it.


@Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
Client. ES promises to maintain compatibility of it with any newer minor
version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
etc.

ES provides also a low level rest client which does not include any
direct es dependencies and can work with any version of ES. It does not
provide any marshalling unmarshalling or higher level features as
Chesnay said.

Correct me if I am wrong @Itamar but your HTTP client is a simplified
version of the ES's high level rest client with a subset of its
features. I think it will still have the same problems as ES's High
Level Rest Client's because ES does not guarantee that newer message
formats will be compatible with older versions of ES or that message
formats are compatible across major versions at all.


@Stephan @Danny As for the 5.x connector. Any ideas how can we get
user's feedback about it? I cross posted on the user mailing list with
no luck so far. Personally I would be in favor of dropping the
connector. Worst case scenario users still have the possibility of
building the connector themselves from source with just bumping the
flink's versions. As far as I can tell there were no changes to the code
base for quite some time.

Best,

Dawid

On 11/02/2020 10:46, Chesnay Schepler wrote:

I suppose the downside in an HTTP ES sink is that you don't get _any_
form of high-level API from ES, and we'd have to manually build an
HTTP request that matches the ES format. Of course you also lose any
client-side verification that the clients did, if there is any (but I
guess the API itself prevented certain errors).

On 11/02/2020 09:32, Stephan Ewen wrote:

+1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
for that one).

@Itamar - if you would be interested in contributing a "universal" or
"cross version" ES connector, that could be very interesting. Do you
know
if there are known performance issues or feature restrictions with that
approach?
@dawid what do you think about that?


On Tue, Feb 11, 2020 at 6:28 AM Danny Chan 

wrote:



5.x seems to have a lot of users, is the 6.x completely compatible with
5.x ~

Best,
Danny Chan
在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
，写道：

Hi all,

As described in this

https://issues.apache.org/jira/browse/FLINK-11720

ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our
e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced
in 5.2.

Both versions are already long eol:

https://www.elastic.co/support/eol


I suggest to drop both connectors 5.x and 2.x. If it is too much to
drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

[jira] [Created] (FLINK-16145) ScheduledUnit toString method throw NPE when Execution is null

[YufeiLiu (Jira)]

YufeiLiu created FLINK-16145:


 Summary: ScheduledUnit toString method throw NPE when Execution is 
null
 Key: FLINK-16145
 URL: https://issues.apache.org/jira/browse/FLINK-16145
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: YufeiLiu
 Attachments: image-2020-02-18-19-58-38-506.png

 !image-2020-02-18-19-58-38-506.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16144) Add client.timeout setting and use that for CLI operations

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16144:


 Summary: Add client.timeout setting and use that for CLI operations
 Key: FLINK-16144
 URL: https://issues.apache.org/jira/browse/FLINK-16144
 Project: Flink
  Issue Type: Improvement
  Components: Command Line Client
Reporter: Aljoscha Krettek
 Fix For: 1.11.0


Currently, the Cli uses the {{akka.client.timeout}} setting. This has 
historical reasons but can be very confusing for users. We should introduce a 
new setting {{client.timeout}} that is used for the client, with a fallback to 
the previous {{akka.client.timeout}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16143) Turn on more date time functions of blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16143:
-

 Summary: Turn on more date time functions of blink planner
 Key: FLINK-16143
 URL: https://issues.apache.org/jira/browse/FLINK-16143
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0


Currently blink planner has a series of built-in function such as

DATEDIFF
DATE_ADD
DATE_SUB

which doesn't get into used. We could add the necessary register, generate and 
convert code to make it available in production scope.

 

what do you think [~jark]?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16142) Memory Leak causes Metaspace OOM error on repeated job submission

[Thomas Wozniakowski (Jira)]

Thomas Wozniakowski created FLINK-16142:
---

 Summary: Memory Leak causes Metaspace OOM error on repeated job 
submission
 Key: FLINK-16142
 URL: https://issues.apache.org/jira/browse/FLINK-16142
 Project: Flink
  Issue Type: Bug
  Components: Client / Job Submission
Affects Versions: 1.10.0
Reporter: Thomas Wozniakowski


Hi Guys,

We've just tried deploying 1.10.0 as it has lots of shiny stuff that fits our 
use-case exactly (RocksDB state backend running in a containerised cluster). 
Unfortunately, it seems like there is a memory leak somewhere in the job 
submission logic. We are getting this error:


{code:java}
2020-02-18 10:22:10,020 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - OPERATOR_NAME switched 
from RUNNING to FAILED.
java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468i
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at 
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at 
org.apache.flink.kinesis.shaded.com.amazon!ws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:398)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.metrics.AwsSdkMetrics.(AwsSdkMetrics.java:359)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:728)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:660)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:652)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:611)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:606)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeListShards(AmazonKinesisClient.java:1534)
at 
org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.listShards(AmazonKinesisClient.java:1528)
at 
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.listShards(KinesisProxy.java:439)
at 
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:389)
at 
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:279)
at 
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:686)
at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:287)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
{code}

(The only change in the above text is the OPERATOR_NAME text where I removed 
some of the internal specifics of our system).

This will reliably happen on a fresh cluster after submitting and cancelling 
our job 3 times.

We are using the presto-s3 plugin, the CEP library and the Kinesis connector.

Please let me know what other diagnostics would be useful.

Tom



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-75: Flink Web UI Improvement Proposal

[Piotr Nowojski]

Hi,

+1 (binding) for FLIP-98
+1 (binding) for FLIP-103 as well. I second Xintong's worries about searching 
for something in a large log file, however I agree this probably could be 
expanded/improved in the future.

Piotrek

> On 14 Feb 2020, at 03:28, Yadong Xie  wrote:
> 
> Hi all,
> 
> I would like to start the vote for FLIP-75 which proposes to provide a
> better experience for Flink users in web UI.
> 
> In order to make the vote easier, the FLIP-75 was split
> into several top-level sub FLIPs:
> 
>   - FLIP-98: Better Back Pressure Detection
>   
> 
>   - FLIP-99: Make Max Exception Configurable
>   
> 
>   - FLIP-100: Add Attempt Information
>   
> 
>   - FLIP-101: Add Pending Slots Detail
>   
> 
>   - FLIP-102: Add More Metrics to TaskManager
>   
> 
>   - FLIP-103: Better TM/JM Log Display
>   
>   - FLIP-104: Add More Metrics to JobManager
>   
> 
> 
> 
> and in order to help everyone better understand the proposal, we spent some
> efforts on making an online POC
> 
> previous web: http://101.132.122.69:8081/
> POC web: http://101.132.122.69:8081/web/index.html
> 
> Please fill in the following format when voting:
> 
> FLIP-98: +1 (or -1)
> FLIP-99: +1 (or -1)
> FLIP-100: +1 (or -1)
> FLIP-101: +1 (or -1)
> FLIP-102: +1 (or -1)
> FLIP-103: +1 (or -1)
> FLIP-104: +1 (or -1)
> 
> The vote will last for at least 72 hours, following the consensus voting
> process.
> 
> FLIP wiki:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-75%3A+Flink+Web+UI+Improvement+Proposal
> 
> Discussion thread:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> 
> Thanks,
> 
> Yadong

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

[Piotr Nowojski]

Hi,

A quick question/comment about FLIP-102. Have you thought about adding GC 
stats? I’m not sure what’s easily do-able, but something that would allow user 
to see GC issues (long/frequent pauses, lots of CPU time spent in the GC) would 
be quite useful for analysing performance/stability issues, without a need of 
connecting profilers in a distributed environment?

Piotrek

> On 10 Feb 2020, at 10:58, Yadong Xie  wrote:
> 
> Hi all
> I have drafted the docs of top-level FLIPs for the individual changes
> proposed in FLIP-75.
> will update it to the cwiki page and start the voting stage soon if there
> is no objection.
> 
>   - FLIP-98: Better Back Pressure Detection
>   
> 
>   - FLIP-99: Make Max Exception Configurable
>   
> 
>   - FLIP-100: Add Attempt Information
>   
> 
>   - FLIP-101: Add Pending Slots Tab in Job Detail
>   
> 
>   - FLIP-102: Add More Metrics to TaskManager
>   
> 
>   - FLIP-103: Better Taskmanager Log Display
>   
> 
>   - FLIP-104: Add More Metrics to Jobmanager
>   
> 
>   - FLIP-105: Better Jobmanager Log Display
>   
> 
> 
> 
> Yadong Xie  于2020年2月9日周日 下午7:24写道：
> 
>> Hi Till
>> I got your point, will create sub FLIPs and votings according to the
>> FLIP-75 and previous discussion soon.
>> 
>> Till Rohrmann  于2020年2月9日周日 下午5:27写道：
>> 
>>> Hi Yadong,
>>> 
>>> I think it would be fine to simply link to this discussion thread to keep
>>> the discussion history. Maybe an easier way would be to create top-level
>>> FLIPs for the individual changes proposed in FLIP-75. The reason I'm
>>> proposing this is that it would be easier to vote on it and to implement
>>> it
>>> because the scope is smaller. But maybe I'm wrong here and others could
>>> chime in to voice their opinion.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Fri, Feb 7, 2020 at 9:58 AM Yadong Xie  wrote:
>>> 
 Hi Till
 
 FLIP-75 has been open since September, and the design doc has been
>>> iterated
 over 3 versions and more than 20 patches.
 I had a try, but it is hard to split the design docs into sub FLIP and
>>> keep
 all the discussion history at the same time.
 
 Maybe it is better to start another discussion to talk about the
>>> individual
 sub FLIP voting? and make the next FLIP follow the new practice if
 possible.
 
 Till Rohrmann  于2020年2月3日周一 下午6:28写道：
 
> I think there is no such description because we never did it before. I
 just
> figured that FLIP-75 could actually be a good candidate to start this
> practice. We would need a community discussion first, though.
> 
> Cheers,
> Till
> 
> On Mon, Feb 3, 2020 at 10:28 AM Yadong Xie 
>>> wrote:
> 
>> Hi Till
>> I didn’t find how to create of sub flip at cwiki.apache.org
>> do you mean to create 9 more FLIPS instead of FLIP-75?
>> 
>> Till Rohrmann  于2020年1月30日周四 下午11:12写道：
>> 
>>> Would it be easier if FLIP-75 would be the umbrella FLIP and we
>>> would
>> vote
>>> on the individual improvements as sub FLIPs? Decreasing the scope
> should
>>> make things easier.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Thu, Jan 30, 2020 at 2:35 PM Robert Metzger <
>>> rmetz...@apache.org>
>>> wrote:
>>> 
 Thanks a lot for this work! I believe the web UI is very
>>> important,
> in
 particular to new users. I'm very happy to see that you are
>>> putting
>>> effort
 into improving the visibility into Flink through the proposed
> changes.
 
 I can not judge if all the changes make total sense, but the
> discussion
>>> has
 been open since September, and a good number of people have
 commented
>> in
 the document.
 I wonder if we can move this FLIP to the VOTing stage?
 
 On Wed, Jan 22, 2020 at 6:27 PM Till Rohrmann <
 trohrm...@apache.org>
 wrote:
 
> Thanks for the update Yadong. Big +1 for the proposed
 improvements
>> for
> Flink's web UI. I think they will be super helpful for our
>>> users.
> 
> Cheers,
> Till
> 
> On Tue, Jan 7, 2020 at 10:00 AM Yadong Xie <

[jira] [Created] (FLINK-16141) HiveTableSourceTest unstable

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-16141:


 Summary: HiveTableSourceTest unstable
 Key: FLINK-16141
 URL: https://issues.apache.org/jira/browse/FLINK-16141
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.10.0
Reporter: Stephan Ewen
 Fix For: 1.11.0


The test fails with various exceptions.
See https://api.travis-ci.org/v3/job/651866181/log.txt



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16140) Translate "Event Processing (CEP)" page into Chinese

[shuai.xu (Jira)]

shuai.xu created FLINK-16140:


 Summary: Translate "Event Processing (CEP)" page into Chinese
 Key: FLINK-16140
 URL: https://issues.apache.org/jira/browse/FLINK-16140
 Project: Flink
  Issue Type: Task
  Components: chinese-translation, Documentation
Affects Versions: 1.10.0
Reporter: shuai.xu


Translate the internal page 
"[https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/cep.html]"; 
into Chinese.

The doc located in "flink/docs/dev/libs/cep.zh.md"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16139) Co-location constraints are not reset on task recovery in DefaultScheduler

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-16139:
---

 Summary: Co-location constraints are not reset on task recovery in 
DefaultScheduler
 Key: FLINK-16139
 URL: https://issues.apache.org/jira/browse/FLINK-16139
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu
Assignee: Zhu Zhu
 Fix For: 1.10.1, 1.11.0


The colocation constraints are not reset on task recovery, which may lead to 
task recovery failures when allocating slots.
We should reset the colocation constraints before resetting vertices, just like 
what we do in the legacy scheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16138) Translate "Overview" page of "DataStream API" into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16138:
---

 Summary: Translate "Overview" page of "DataStream API" into Chinese
 Key: FLINK-16138
 URL: https://issues.apache.org/jira/browse/FLINK-16138
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


The page is located at _flink/docs/dev/datastream_api.md_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16137) Translate all DataStream API related pages into Chinese

[Yun Gao (Jira)]

Yun Gao created FLINK-16137:
---

 Summary: Translate all DataStream API related pages into Chinese 
 Key: FLINK-16137
 URL: https://issues.apache.org/jira/browse/FLINK-16137
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation, Documentation
Reporter: Yun Gao
 Fix For: 1.11.0


Translate data stream related pages into Chinese, including the pages under the 
section "Application Development/Streaming(DataStream API)" in the document 
site. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16136) YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority() fails

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16136:
--

 Summary: 
YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority()
 fails
 Key: FLINK-16136
 URL: https://issues.apache.org/jira/browse/FLINK-16136
 Project: Flink
  Issue Type: Task
  Components: Deployment / YARN, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger


https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5259&view=logs&j=764762df-f65b-572b-3d5c-65518c777be4&t=da3c2718-4b76-56bf-ef25-cd33ea381f78
https://transfer.sh/G229r/20200218.1.tar.gz

{code:java}
01:29:45,184 [main] INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Notifying 
ContainerManager to unblock new container-requests
01:29:45,184 [ResourceManager Event Processor] INFO  
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
  - Added node 8b508210238a:37199 clusterResource: 
01:29:45,192 [main] INFO  
org.apache.hadoop.yarn.server.MiniYARNCluster - All Node 
Managers connected in MiniYARNCluster
01:29:45,192 [AsyncDispatcher event handler] INFO  
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl  - Node 
8b508210238a:37199 reported UNHEALTHY with details: 1/1 local-dirs are bad: 
/__w/1/s/flink-yarn-tests/target/YarnTest_3d15de1b-4c9a-411b-81b1-3ca2788b479a/YarnTest_3d15de1b-4c9a-411b-81b1-3ca2788b479a-localDir-nm-1_0;
 1/1 log-dirs are bad: 
/__w/1/s/flink-yarn-tests/target/YarnTest_3d15de1b-4c9a-411b-81b1-3ca2788b479a/YarnTest_3d15de1b-4c9a-411b-81b1-3ca2788b479a-logDir-nm-1_0
01:29:45,192 [AsyncDispatcher event handler] INFO  
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl  - 
8b508210238a:37199 Node Transitioned from RUNNING to UNHEALTHY
01:29:45,193 [ResourceManager Event Processor] INFO  
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
  - Removed node 8b508210238a:37199 clusterResource: 


Test 
yarnApplication_submissionWithPriority_shouldRespectPriority(org.apache.flink.yarn.YarnPrioritySchedulingITCase)
 is running.

01:29:45,449 [main] INFO  org.apache.hadoop.yarn.client.RMProxy 
- Connecting to ResourceManager at 
8b508210238a/172.18.0.2:40977
01:29:45,529 [main] INFO  org.apache.flink.yarn.YarnTestBase
- Running with args [-j, 
/__w/1/s/flink-yarn-tests/../build-target/lib/flink-dist_2.11-1.11-SNAPSHOT.jar,
 -t, /__w/1/s/flink-yarn-tests/../build-target/lib, -t, 
/__w/1/s/flink-yarn-tests/target/shaded-hadoop, -jm, 768m, -tm, 1024m, 
-Dyarn.application.priority=5]
01:29:45,555 [Frontend (CLI/YARN Client) runner thread (startWithArgs()).] WARN 
 org.apache.flink.yarn.cli.FlinkYarnSessionCli - The 
configuration directory ('/tmp/junit3457413859979027256/conf') already contains 
a LOG4J config file.If you want to use logback, then please delete or rename 
the log configuration file.
01:29:45,567 [Frontend (CLI/YARN Client) runner thread (startWithArgs()).] INFO 
 org.apache.flink.yarn.cli.FlinkYarnSessionCli - Dynamic 
Property set: yarn.application.priority=5
01:29:45,591 [Frontend (CLI/YARN Client) runner thread (startWithArgs()).] INFO 
 org.apache.hadoop.yarn.client.RMProxy - Connecting to 
ResourceManager at 8b508210238a/172.18.0.2:40977
01:29:45,611 [Frontend (CLI/YARN Client) runner thread (startWithArgs()).] INFO 
 org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils  - The 
derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less 
than its min value 192.000mb (201326592 bytes), min value will be used instead
01:29:45,753 [Frontend (CLI/YARN Client) runner thread (startWithArgs()).] INFO 
 org.apache.flink.yarn.YarnTestBase- Runner stopped 
with exception
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy 
Yarn session cluster
at 
org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:381)
at 
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:548)
at org.apache.flink.yarn.YarnTestBase$Runner.run(YarnTestBase.java:917)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The 
number of requested virtual cores for application master 1 exceeds the maximum 
number of virtual cores 0 available in the Yarn Cluster.
at 
org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:284)
at 
org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:444)
at 
org.apache.flink.yar

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Robert Metzger]

The ES5 connector is causing some problems on the CI system. It would be
nice if we could make a decision here soon. I don't want to invest time
into fixing it, if we are going to remove it.

I'm still in favor of removing it. If we see that there's demand for the
5.x connector after the 1.11 release, somebody can take the source and
contribute it to Apache Bahir or a GitHub account and then posts it to
flink-packages.org.


On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz 
wrote:

> Sorry for late reply,
>
> @all I think there is a general consensus that we want to drop ES 2.x
> support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
> track it.
>
>
> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
> Client. ES promises to maintain compatibility of it with any newer minor
> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
> etc.
>
> ES provides also a low level rest client which does not include any
> direct es dependencies and can work with any version of ES. It does not
> provide any marshalling unmarshalling or higher level features as
> Chesnay said.
>
> Correct me if I am wrong @Itamar but your HTTP client is a simplified
> version of the ES's high level rest client with a subset of its
> features. I think it will still have the same problems as ES's High
> Level Rest Client's because ES does not guarantee that newer message
> formats will be compatible with older versions of ES or that message
> formats are compatible across major versions at all.
>
>
> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
> user's feedback about it? I cross posted on the user mailing list with
> no luck so far. Personally I would be in favor of dropping the
> connector. Worst case scenario users still have the possibility of
> building the connector themselves from source with just bumping the
> flink's versions. As far as I can tell there were no changes to the code
> base for quite some time.
>
> Best,
>
> Dawid
>
> On 11/02/2020 10:46, Chesnay Schepler wrote:
> > I suppose the downside in an HTTP ES sink is that you don't get _any_
> > form of high-level API from ES, and we'd have to manually build an
> > HTTP request that matches the ES format. Of course you also lose any
> > client-side verification that the clients did, if there is any (but I
> > guess the API itself prevented certain errors).
> >
> > On 11/02/2020 09:32, Stephan Ewen wrote:
> >> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
> >> for that one).
> >>
> >> @Itamar - if you would be interested in contributing a "universal" or
> >> "cross version" ES connector, that could be very interesting. Do you
> >> know
> >> if there are known performance issues or feature restrictions with that
> >> approach?
> >> @dawid what do you think about that?
> >>
> >>
> >> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan 
> wrote:
> >>
> >>> 5.x seems to have a lot of users, is the 6.x completely compatible with
> >>> 5.x ~
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
> >>> ，写道：
>  Hi all,
> 
>  As described in this
> https://issues.apache.org/jira/browse/FLINK-11720
>  ticket our elasticsearch 5.x connector does not work out of the box on
>  some systems and requires a version bump. This also happens for our
>  e2e.
>  We cannot bump the version in es 5.x connector, because 5.x connector
>  shares a common class with 2.x that uses an API that was replaced
>  in 5.2.
> 
>  Both versions are already long eol:
> https://www.elastic.co/support/eol
> 
>  I suggest to drop both connectors 5.x and 2.x. If it is too much to
>  drop
>  both of them, I would strongly suggest dropping at least 2.x connector
>  and update the 5.x line to a working es client module.
> 
>  What do you think? Should we drop both versions? Drop only the 2.x
>  connector? Or keep them both?
> 
>  Best,
> 
>  Dawid
> 
> 
> >
>
>

Sharing operator subtask state using side outputs

[vishalovercome]

I am implementing a streaming application and one of the stateful operators
is trying to capture a “owner has items” relationship. The state, keyed per
owner consists of details about the owner and each of the items. Ownership
of an item can change and I would like to be able associate each item to its
correct owner. Since operator state for different owners could be in
different subtasks and these subtasks are intended to operate independently,
I want to know what’s the best way to share state. One solution that I was
able to think of was to create a keyed datastream from the side output of a
subtask and have it sent to the correct owner and clear the state from the
original owner. Essentially:
1. Subtask1 with state about OldOwner that has Item1, Item2, … , ItemN
2. Subtask1 writes to a message to side output (OldOwner, NewOwner,
List[ItemsToTransfer])
3. (Optional) Clear state about List[ItemsToTransfer] from state about
OldOwner.
4. Create a datastream from the side output and send it back to the same
operator, but potentially different subtask that has state about NewOwner.
5. Update the state of NewOwner by adding the new set of items

Since side outputs are intended for a very different purpose (logging,
etc.), I want to know whether this will work. Do the same fault tolerance
guarantees apply to side outputs as to regular data streams? Is there a
limit to how many side output messages can be buffered in a subtask? 

An alternative approach might be to take the output of the first subtask and
feed it back to the same operator. Both these approaches, in theory violate
the property that a flink job is a DAG, although for my use case, there
would never be a cyclic data transfer. 



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-16135) Unify AutoContextClassLoader and TemporaryClassLoaderContext

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-16135:


 Summary: Unify AutoContextClassLoader and 
TemporaryClassLoaderContext
 Key: FLINK-16135
 URL: https://issues.apache.org/jira/browse/FLINK-16135
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.11.0


AutoContextClassLoader and TemporaryClassLoaderContext do the same thing. We 
should consolidate all uses to one of them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16134) npm ERR! 429 Too Many Requests

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-16134:
--

 Summary: npm ERR! 429 Too Many Requests
 Key: FLINK-16134
 URL: https://issues.apache.org/jira/browse/FLINK-16134
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: Piotr Nowojski


Test of {{flink-runtime-web}} failed with:

{noformat}
[INFO] Installing node version v10.9.0
[INFO] Downloading 
https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.gz to 
/home/agent1_azpcontainer/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz
[INFO] No proxies configured
[INFO] No proxy was configured, downloading directly
[INFO] Unpacking 
/home/agent1_azpcontainer/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz
 into /__w/2/s/flink-runtime-web/web-dashboard/node/tmp
[INFO] Copying node binary from 
/__w/2/s/flink-runtime-web/web-dashboard/node/tmp/node-v10.9.0-linux-x64/bin/node
 to /__w/2/s/flink-runtime-web/web-dashboard/node/node
[INFO] Extracting NPM
[INFO] Installed node locally.
[INFO] 
[INFO] --- frontend-maven-plugin:1.6:npm (npm install) @ flink-runtime-web_2.11 
---
[INFO] Running 'npm ci --cache-max=0 --no-save' in 
/__w/2/s/flink-runtime-web/web-dashboard
[ERROR] npm ERR! code E429
[ERROR] npm ERR! 429 Too Many Requests: ts-node@7.0.1
[ERROR] 
[ERROR] npm ERR! A complete log of this run can be found in:
[ERROR] npm ERR! 
/home/agent1_azpcontainer/.npm/_logs/2020-02-17T12_31_39_912Z-debug.log
{noformat}

https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/5244/logs/8





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-84: Improve & Refactor execute/sqlQuery/sqlUpdate APIS of TableEnvironment

[godfrey he]

Thanks Kurt and Jark for explanation, I now also think we should make the
TableEnvironment interface more statable and should not change "sqlQuery"
method and "from" method.

Hi Jingsong. Regarding to the "DmlBatch", I totally agree with advantages
of "addBatch" method. However, there are two more questions need to solve:
one is how users write multi-sink programs in a Table API ? and another is
how users explain multi-sink program in both SQL and Table API ?
Currently, "DmlBatch" class can solve those questions. (the main
disadvantages is Inconsistent with the current interface)

Bests,
godfrey

Jingsong Li  于2020年2月15日周六 下午9:09写道：

> Hi Kurt and Godfrey,
>
> Thank you for your explanation.
>
> Regarding to the "DmlBatch",
> I see there are some description for JDBC Statement.addBatch in the
> document.
> What do you think about introducing "addBatch" to the TableEnv instead of
> introducing a new class?
> The advantage is:
> - Consistent with JDBC statement.
> - Consistent with current interface, what we need do is just modify method
> name.
>
> Best,
> Jingsong Lee
>
>
> On Sat, Feb 15, 2020 at 4:48 PM Kurt Young  wrote:
>
> > I don't think we should change `from` to `fromCatalog`, especially `from`
> > is just
> > introduced in 1.10. I agree with Jark we should change interface only
> when
> > necessary,
> > e.g. the semantic is broken or confusing. So I'm +1 to keep `sqlQuery` as
> > it is.
> >
> > Best,
> > Kurt
> >
> >
> > On Sat, Feb 15, 2020 at 3:59 PM Jark Wu  wrote:
> >
> > > Thanks Kurt and Godfrey for the explanation,
> > >
> > > It makes sense to me that renaming `from(tableName)` to
> > > `fromCatalog(tableName)`.
> > > However, I still think `sqlQuery(query)` is clear and works well. Is it
> > > necessary to change it?
> > >
> > > We removed `sql(query)` and introduced `sqlQuery(query)`, we removed
> > > `scan(tableName)` and introduced `from(tableName)`,
> > > and now we want to remove them again. Users will feel like the
> interface
> > is
> > > very unstable, that really frustrates users.
> > > I think we should be cautious to remove interface and only when it is
> > > necessary.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > >
> > > On Thu, 13 Feb 2020 at 20:58, godfrey he  wrote:
> > >
> > > > hi kurt，jark，jingsong
> > > >
> > > > Regarding to "fromQuery", I agree with kurt. In addition, I think
> > `Table
> > > > from(String tableName)` should be renamed to `Table
> fromCatalog(String
> > > > tableName)`.
> > > >
> > > > Regarding to the "DmlBatch", DML contains "INSERT", "UPDATE",
> "DELETE",
> > > and
> > > > they can be executed in a same batch in the future. So we can add
> > > > "addUpdate" method and "addDelete" method to support them.
> > > >
> > > > Regarding to the "Inserts addInsert", maybe we can add a
> > > "DmlBatchBuilder".
> > > >
> > > > open to more discussion
> > > >
> > > > Best,
> > > > godfrey
> > > >
> > > >
> > > >
> > > > Kurt Young  于2020年2月13日周四 下午4:56写道：
> > > >
> > > > > Regarding to "fromQuery" is confusing users with "Table from(String
> > > > > tableName)", I have
> > > > > a just opposite opinion. I think this "fromXXX" pattern can make
> > users
> > > > > quite clear when they
> > > > > want to get a Table from TableEnvironment. Similar interfaces will
> > also
> > > > > include like "fromElements".
> > > > >
> > > > > Regarding to the name of DmlBatch, I think it's mainly for
> > > > > future flexibility, in case we can support
> > > > > other statement in a single batch. If that happens, the name
> > "Inserts"
> > > > will
> > > > > be weird.
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Thu, Feb 13, 2020 at 4:03 PM Jark Wu  wrote:
> > > > >
> > > > > > I agree with Jingsong.
> > > > > >
> > > > > > +1 to keep `sqlQuery`, it's clear from the method name and return
> > > type
> > > > > that
> > > > > > it accepts a SELECT query and returns a logic representation
> > `Table`.
> > > > > > The `fromQuery` is a little confused users with the `Table
> > > from(String
> > > > > > tableName)` method.
> > > > > >
> > > > > > Regarding to the `DmlBatch`, I agree with Jingsong, AFAIK, the
> > > purpose
> > > > of
> > > > > > `DmlBatch` is used to batching insert statements.
> > > > > > Besides, DML terminology is not commonly know among users. So
> what
> > > > about
> > > > > > `InsertsBatching startBatchingInserts()` ?
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Thu, 13 Feb 2020 at 15:50, Jingsong Li <
> jingsongl...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Godfrey,
> > > > > > >
> > > > > > > Thanks for updating. +1 sketchy.
> > > > > > >
> > > > > > > I have no idea to change "sqlQuery" to "fromQuery", I think
> > > > "sqlQuery"
> > > > > is
> > > > > > > OK, It's not that confusing with return values.
> > > > > > >
> > > > > > > Can we change the "DmlBatch" to "Inserts"?  I don't see any
> other
> > > > > needs.
> > > > > > > "Dml" seems a little weird.
> > > > > > > It is better to su

[DISCUSS] Improvements on FLIP Process

[Hequn Cheng]

Hi everyone,

Currently, when we create a FLIP we follow the FLIP process in the Flink
Improvement Proposals wiki[1]. The process mainly includes the following
steps:
1. Create a FLIP wiki page.
2. Raise the discussion on the mailing list.
3. Once the proposal is finalized, call a vote to have the proposal adopted.
There is also a discussion[2] on the FLIP process which may be helpful for
you.

As it is not allowed commented on the wiki, we usually have a google doc
for the discussion at step 2 and whenever there is a change we need to pick
it to the wiki page. This makes things somehow redundant. To solve this, we
can rearrange the step a little bit and avoid the pick:
1. Raise the discussion on the mailing list. The subject of the thread is
of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
should follow the FLIP-Template strictly. (The [FLIP] tag is used to inform
people that it is a FLIP discussion and more attention should be paid.)
2. Create a FLIP wiki page once we reached an agreement on the discussion.
We can simply copy the google doc into the FLIP wiki page.
3. Once the proposal is finalized, call a vote to have the proposal
adopted. It should be noted that we should always vote on a FLIP wiki page
instead of a google doc. The wiki page is the final version of the google
doc.

This can bring some benefits:
1. Make the discussion more effective as we force people to write and
discuss on a google doc that follows the FLIP template which
includes necessary information such as Motivation, Interfaces, Proposed
changes, etc.
2. Avoid redundant pick from google doc to Flink wiki page. Once we reached
an agreement on the discussion, we can simply copy the google doc into the
FLIP wiki page.
3. As adopted FLIP should mostly be "immutable", we can even make the wiki
page PMC or committer editable since it just needs a simple copy from the
google doc.

Looking forward to your feedback!

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-META-FLIP-Sticking-or-not-to-a-strict-FLIP-voting-process-td29978.html#a29988