Re: Change in accumutors semantics with jobClient
My pleasure ! Thanks for pointing out the release notes. Cheers, Etienne On 23/06/2021 14:00, Till Rohrmann wrote: Yes, it should be part of the release notes where this change was introduced. I'll take a look at your PR. Thanks a lot Etienne. Cheers, Till On Wed, Jun 23, 2021 at 12:29 PM Etienne Chauchot wrote: Hi Till, Of course I can update the release notes. Question is: this change is quite old (January), it is already available in all the maintained releases :1.11, 1.12, 1.13. I think I should update the release notes for all these versions no ? In case you agree, I took the liberty to update all these release notes in a PR: https://github.com/apache/flink/pull/16256 Cheers, Etienne On 21/06/2021 11:39, Till Rohrmann wrote: Thanks for bringing this to the dev ML Etienne. Could you maybe update the release notes for Flink 1.13 [1] to include this change? That way it might be a bit more prominent. I think the change needs to go into the release-1.13 and master branch. [1] https://github.com/apache/flink/blob/master/docs/content/release-notes/flink-1.13.md Cheers, Till On Fri, Jun 18, 2021 at 2:45 PM Etienne Chauchot wrote: Hi all, I did a fix some time ago regarding accumulators: the/JobClient.getAccumulators()/ was infinitely blocking in local environment for a streaming job (1). The change (2) consisted of giving the current accumulators value for the running job. And when fixing this in the PR, it appeared that I had to change the accumulators semantics with /JobClient/ and I just realized that I forgot to bring this back to the ML: Previously /JobClient/ assumed that getAccumulator() was called on a bounded pipeline and that the user wanted to acquire the *final accumulator values* after the job is finished. But now it returns the *current value of accumulators* immediately to be compatible with unbounded pipelines. If it is run on a bounded pipeline, then to get the final accumulator values after the job is finished, one needs to call /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ (1): https://issues.apache.org/jira/browse/FLINK-18685 (2): https://github.com/apache/flink/pull/14558# Cheers, Etienne
Re: Change in accumutors semantics with jobClient
Yes, it should be part of the release notes where this change was introduced. I'll take a look at your PR. Thanks a lot Etienne. Cheers, Till On Wed, Jun 23, 2021 at 12:29 PM Etienne Chauchot wrote: > Hi Till, > > Of course I can update the release notes. > > Question is: this change is quite old (January), it is already available > in all the maintained releases :1.11, 1.12, 1.13. > > I think I should update the release notes for all these versions no ? > > In case you agree, I took the liberty to update all these release notes > in a PR: https://github.com/apache/flink/pull/16256 > > Cheers, > > Etienne > > On 21/06/2021 11:39, Till Rohrmann wrote: > > Thanks for bringing this to the dev ML Etienne. Could you maybe update > the > > release notes for Flink 1.13 [1] to include this change? That way it > might > > be a bit more prominent. I think the change needs to go into the > > release-1.13 and master branch. > > > > [1] > > > https://github.com/apache/flink/blob/master/docs/content/release-notes/flink-1.13.md > > > > Cheers, > > Till > > > > > > On Fri, Jun 18, 2021 at 2:45 PM Etienne Chauchot > > wrote: > > > >> Hi all, > >> > >> I did a fix some time ago regarding accumulators: > >> the/JobClient.getAccumulators()/ was infinitely blocking in local > >> environment for a streaming job (1). The change (2) consisted of giving > >> the current accumulators value for the running job. And when fixing this > >> in the PR, it appeared that I had to change the accumulators semantics > >> with /JobClient/ and I just realized that I forgot to bring this back to > >> the ML: > >> > >> Previously /JobClient/ assumed that getAccumulator() was called on a > >> bounded pipeline and that the user wanted to acquire the *final > >> accumulator values* after the job is finished. > >> > >> But now it returns the *current value of accumulators* immediately to be > >> compatible with unbounded pipelines. > >> > >> If it is run on a bounded pipeline, then to get the final accumulator > >> values after the job is finished, one needs to call > >> > >> > /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ > >> > >> (1): https://issues.apache.org/jira/browse/FLINK-18685 > >> > >> (2): https://github.com/apache/flink/pull/14558# > >> > >> > >> Cheers, > >> > >> Etienne > >> > >> >
Re: Change in accumutors semantics with jobClient
Hi Till, Of course I can update the release notes. Question is: this change is quite old (January), it is already available in all the maintained releases :1.11, 1.12, 1.13. I think I should update the release notes for all these versions no ? In case you agree, I took the liberty to update all these release notes in a PR: https://github.com/apache/flink/pull/16256 Cheers, Etienne On 21/06/2021 11:39, Till Rohrmann wrote: Thanks for bringing this to the dev ML Etienne. Could you maybe update the release notes for Flink 1.13 [1] to include this change? That way it might be a bit more prominent. I think the change needs to go into the release-1.13 and master branch. [1] https://github.com/apache/flink/blob/master/docs/content/release-notes/flink-1.13.md Cheers, Till On Fri, Jun 18, 2021 at 2:45 PM Etienne Chauchot wrote: Hi all, I did a fix some time ago regarding accumulators: the/JobClient.getAccumulators()/ was infinitely blocking in local environment for a streaming job (1). The change (2) consisted of giving the current accumulators value for the running job. And when fixing this in the PR, it appeared that I had to change the accumulators semantics with /JobClient/ and I just realized that I forgot to bring this back to the ML: Previously /JobClient/ assumed that getAccumulator() was called on a bounded pipeline and that the user wanted to acquire the *final accumulator values* after the job is finished. But now it returns the *current value of accumulators* immediately to be compatible with unbounded pipelines. If it is run on a bounded pipeline, then to get the final accumulator values after the job is finished, one needs to call /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ (1): https://issues.apache.org/jira/browse/FLINK-18685 (2): https://github.com/apache/flink/pull/14558# Cheers, Etienne
Re: Change in accumutors semantics with jobClient
Thanks for bringing this to the dev ML Etienne. Could you maybe update the release notes for Flink 1.13 [1] to include this change? That way it might be a bit more prominent. I think the change needs to go into the release-1.13 and master branch. [1] https://github.com/apache/flink/blob/master/docs/content/release-notes/flink-1.13.md Cheers, Till On Fri, Jun 18, 2021 at 2:45 PM Etienne Chauchot wrote: > Hi all, > > I did a fix some time ago regarding accumulators: > the/JobClient.getAccumulators()/ was infinitely blocking in local > environment for a streaming job (1). The change (2) consisted of giving > the current accumulators value for the running job. And when fixing this > in the PR, it appeared that I had to change the accumulators semantics > with /JobClient/ and I just realized that I forgot to bring this back to > the ML: > > Previously /JobClient/ assumed that getAccumulator() was called on a > bounded pipeline and that the user wanted to acquire the *final > accumulator values* after the job is finished. > > But now it returns the *current value of accumulators* immediately to be > compatible with unbounded pipelines. > > If it is run on a bounded pipeline, then to get the final accumulator > values after the job is finished, one needs to call > > /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ > > (1): https://issues.apache.org/jira/browse/FLINK-18685 > > (2): https://github.com/apache/flink/pull/14558# > > > Cheers, > > Etienne > >
Change in accumutors semantics with jobClient
Hi all, I did a fix some time ago regarding accumulators: the/JobClient.getAccumulators()/ was infinitely blocking in local environment for a streaming job (1). The change (2) consisted of giving the current accumulators value for the running job. And when fixing this in the PR, it appeared that I had to change the accumulators semantics with /JobClient/ and I just realized that I forgot to bring this back to the ML: Previously /JobClient/ assumed that getAccumulator() was called on a bounded pipeline and that the user wanted to acquire the *final accumulator values* after the job is finished. But now it returns the *current value of accumulators* immediately to be compatible with unbounded pipelines. If it is run on a bounded pipeline, then to get the final accumulator values after the job is finished, one needs to call /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ (1): https://issues.apache.org/jira/browse/FLINK-18685 (2): https://github.com/apache/flink/pull/14558# Cheers, Etienne
