Hi Thomas, Thanks a lot for the detailed information.
I think the problem is in CheckpointCoordinator. It stores the last checkpoint completion time after checking queued requests. I've created a ticket to fix this: https://issues.apache.org/jira/browse/FLINK-18856 On Sat, Aug 8, 2020 at 5:25 AM Thomas Weise <t...@apache.org> wrote: > Just another update: > > The duration of snapshotState is capped by the Kinesis > producer's "RecordTtl" setting (default 30s). The sleep time in flushSync > does not contribute to the observed behavior. > > I guess the open question is why, with the same settings, is 1.11 since > commit 355184d69a8519d29937725c8d85e8465d7e3a90 processing more checkpoints? > > > On Fri, Aug 7, 2020 at 9:15 AM Thomas Weise <t...@apache.org> wrote: > >> Hi Roman, >> >> Here are the checkpoint summaries for both commits: >> >> >> https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit#slide=id.g86d15b2fc7_0_0 >> >> The config: >> >> CheckpointConfig checkpointConfig = env.getCheckpointConfig(); >> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); >> checkpointConfig.setCheckpointInterval(*10_000*); >> checkpointConfig.setMinPauseBetweenCheckpoints(*10_000*); >> >> checkpointConfig.enableExternalizedCheckpoints(DELETE_ON_CANCELLATION); >> checkpointConfig.setCheckpointTimeout(600_000); >> checkpointConfig.setMaxConcurrentCheckpoints(1); >> checkpointConfig.setFailOnCheckpointingErrors(true); >> >> The values marked bold when changed to *60_000* make the symptom >> disappear. I meanwhile also verified that with the 1.11.0 release commit. >> >> I will take a look at the sleep time issue. >> >> Thanks, >> Thomas >> >> >> On Fri, Aug 7, 2020 at 1:44 AM Roman Khachatryan <ro...@data-artisans.com> >> wrote: >> >>> Hi Thomas, >>> >>> Thanks for your reply! >>> >>> I think you are right, we can remove this sleep and improve >>> KinesisProducer. >>> Probably, it's snapshotState can also be sped up by forcing records >>> flush more often. >>> Do you see that 30s checkpointing duration is caused by KinesisProducer >>> (or maybe other operators)? >>> >>> I'd also like to understand the reason behind this increase in >>> checkpoint frequency. >>> Can you please share these values: >>> - execution.checkpointing.min-pause >>> - execution.checkpointing.max-concurrent-checkpoints >>> - execution.checkpointing.timeout >>> >>> And what is the "new" observed checkpoint frequency (or how many >>> checkpoints are created) compared to older versions? >>> >>> >>> On Fri, Aug 7, 2020 at 4:49 AM Thomas Weise <t...@apache.org> wrote: >>> >>>> Hi Roman, >>>> >>>> Indeed there are more frequent checkpoints with this change! The >>>> application was configured to checkpoint every 10s. With 1.10 ("good >>>> commit"), that leads to fewer completed checkpoints compared to 1.11 >>>> ("bad >>>> commit"). Just to be clear, the only difference between the two runs was >>>> the commit 355184d69a8519d29937725c8d85e8465d7e3a90 >>>> >>>> Since the sync part of checkpoints with the Kinesis producer always >>>> takes >>>> ~30 seconds, the 10s configured checkpoint frequency really had no >>>> effect >>>> before 1.11. I confirmed that both commits perform comparably by setting >>>> the checkpoint frequency and min pause to 60s. >>>> >>>> I still have to verify with the final 1.11.0 release commit. >>>> >>>> It's probably good to take a look at the Kinesis producer. Is it really >>>> necessary to have 500ms sleep time? What's responsible for the ~30s >>>> duration in snapshotState? >>>> >>>> As things stand it doesn't make sense to use checkpoint intervals < 30s >>>> when using the Kinesis producer. >>>> >>>> Thanks, >>>> Thomas >>>> >>>> On Sat, Aug 1, 2020 at 2:53 PM Roman Khachatryan < >>>> ro...@data-artisans.com> >>>> wrote: >>>> >>>> > Hi Thomas, >>>> > >>>> > Thanks a lot for the analysis. >>>> > >>>> > The first thing that I'd check is whether checkpoints became more >>>> frequent >>>> > with this commit (as each of them adds at least 500ms if there is at >>>> least >>>> > one not sent record, according to FlinkKinesisProducer.snapshotState). >>>> > >>>> > Can you share checkpointing statistics (1.10 vs 1.11 or last "good" vs >>>> > first "bad" commits)? >>>> > >>>> > On Fri, Jul 31, 2020 at 5:29 AM Thomas Weise <thomas.we...@gmail.com> >>>> > wrote: >>>> > >>>> > > I run git bisect and the first commit that shows the regression is: >>>> > > >>>> > > >>>> > > >>>> > >>>> https://github.com/apache/flink/commit/355184d69a8519d29937725c8d85e8465d7e3a90 >>>> > > >>>> > > >>>> > > On Thu, Jul 23, 2020 at 6:46 PM Kurt Young <ykt...@gmail.com> >>>> wrote: >>>> > > >>>> > > > From my experience, java profilers are sometimes not accurate >>>> enough to >>>> > > > find out the performance regression >>>> > > > root cause. In this case, I would suggest you try out intel vtune >>>> > > amplifier >>>> > > > to watch more detailed metrics. >>>> > > > >>>> > > > Best, >>>> > > > Kurt >>>> > > > >>>> > > > >>>> > > > On Fri, Jul 24, 2020 at 8:51 AM Thomas Weise <t...@apache.org> >>>> wrote: >>>> > > > >>>> > > > > The cause of the issue is all but clear. >>>> > > > > >>>> > > > > Previously I had mentioned that there is no suspect change to >>>> the >>>> > > Kinesis >>>> > > > > connector and that I had reverted the AWS SDK change to no >>>> effect. >>>> > > > > >>>> > > > > https://issues.apache.org/jira/browse/FLINK-17496 actually >>>> fixed >>>> > > another >>>> > > > > regression in the previous release and is present before and >>>> after. >>>> > > > > >>>> > > > > I repeated the run with 1.11.0 core and downgraded the entire >>>> Kinesis >>>> > > > > connector to 1.10.1: Nothing changes, i.e. the regression is >>>> still >>>> > > > present. >>>> > > > > Therefore we will need to look elsewhere for the root cause. >>>> > > > > >>>> > > > > Regarding the time spent in snapshotState, repeat runs reveal a >>>> wide >>>> > > > range >>>> > > > > for both versions, 1.10 and 1.11. So again this is nothing >>>> pointing >>>> > to >>>> > > a >>>> > > > > root cause. >>>> > > > > >>>> > > > > At this point, I have no ideas remaining other than doing a >>>> bisect to >>>> > > > find >>>> > > > > the culprit. Any other suggestions? >>>> > > > > >>>> > > > > Thomas >>>> > > > > >>>> > > > > >>>> > > > > On Thu, Jul 16, 2020 at 9:19 PM Zhijiang < >>>> wangzhijiang...@aliyun.com >>>> > > > > .invalid> >>>> > > > > wrote: >>>> > > > > >>>> > > > > > Hi Thomas, >>>> > > > > > >>>> > > > > > Thanks for your further profiling information and glad to see >>>> we >>>> > > > already >>>> > > > > > finalized the location to cause the regression. >>>> > > > > > Actually I was also suspicious of the point of #snapshotState >>>> in >>>> > > > previous >>>> > > > > > discussions since it indeed cost much time to block normal >>>> operator >>>> > > > > > processing. >>>> > > > > > >>>> > > > > > Based on your below feedback, the sleep time during >>>> #snapshotState >>>> > > > might >>>> > > > > > be the main concern, and I also digged into the >>>> implementation of >>>> > > > > > FlinkKinesisProducer#snapshotState. >>>> > > > > > while (producer.getOutstandingRecordsCount() > 0) { >>>> > > > > > producer.flush(); >>>> > > > > > try { >>>> > > > > > Thread.sleep(500); >>>> > > > > > } catch (InterruptedException e) { >>>> > > > > > LOG.warn("Flushing was interrupted."); >>>> > > > > > break; >>>> > > > > > } >>>> > > > > > } >>>> > > > > > It seems that the sleep time is mainly affected by the >>>> internal >>>> > > > > operations >>>> > > > > > inside KinesisProducer implementation provided by amazonaws, >>>> which >>>> > I >>>> > > am >>>> > > > > not >>>> > > > > > quite familiar with. >>>> > > > > > But I noticed there were two upgrades related to it in >>>> > > release-1.11.0. >>>> > > > > One >>>> > > > > > is for upgrading amazon-kinesis-producer to 0.14.0 [1] and >>>> another >>>> > is >>>> > > > for >>>> > > > > > upgrading aws-sdk-version to 1.11.754 [2]. >>>> > > > > > You mentioned that you already reverted the SDK upgrade to >>>> verify >>>> > no >>>> > > > > > changes. Did you also revert the [1] to verify? >>>> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-17496 >>>> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-14881 >>>> > > > > > >>>> > > > > > Best, >>>> > > > > > Zhijiang >>>> > > > > > >>>> ------------------------------------------------------------------ >>>> > > > > > From:Thomas Weise <t...@apache.org> >>>> > > > > > Send Time:2020年7月17日(星期五) 05:29 >>>> > > > > > To:dev <dev@flink.apache.org> >>>> > > > > > Cc:Zhijiang <wangzhijiang...@aliyun.com>; Stephan Ewen < >>>> > > > se...@apache.org >>>> > > > > >; >>>> > > > > > Arvid Heise <ar...@ververica.com>; Aljoscha Krettek < >>>> > > > aljos...@apache.org >>>> > > > > > >>>> > > > > > Subject:Re: Kinesis Performance Issue (was [VOTE] Release >>>> 1.11.0, >>>> > > > release >>>> > > > > > candidate #4) >>>> > > > > > >>>> > > > > > Sorry for the delay. >>>> > > > > > >>>> > > > > > I confirmed that the regression is due to the sink >>>> (unsurprising, >>>> > > since >>>> > > > > > another job with the same consumer, but not the producer, >>>> runs as >>>> > > > > > expected). >>>> > > > > > >>>> > > > > > As promised I did CPU profiling on the problematic >>>> application, >>>> > which >>>> > > > > gives >>>> > > > > > more insight into the regression [1] >>>> > > > > > >>>> > > > > > The screenshots show that the average time for snapshotState >>>> > > increases >>>> > > > > from >>>> > > > > > ~9s to ~28s. The data also shows the increase in sleep time >>>> during >>>> > > > > > snapshotState. >>>> > > > > > >>>> > > > > > Does anyone, based on changes made in 1.11, have a theory why? >>>> > > > > > >>>> > > > > > I had previously looked at the changes to the Kinesis >>>> connector and >>>> > > > also >>>> > > > > > reverted the SDK upgrade, which did not change the situation. >>>> > > > > > >>>> > > > > > It will likely be necessary to drill into the sink / >>>> checkpointing >>>> > > > > details >>>> > > > > > to understand the cause of the problem. >>>> > > > > > >>>> > > > > > Let me know if anyone has specific questions that I can >>>> answer from >>>> > > the >>>> > > > > > profiling results. >>>> > > > > > >>>> > > > > > Thomas >>>> > > > > > >>>> > > > > > [1] >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit?usp=sharing >>>> > > > > > >>>> > > > > > On Mon, Jul 13, 2020 at 11:14 AM Thomas Weise <t...@apache.org >>>> > >>>> > > wrote: >>>> > > > > > >>>> > > > > > > + dev@ for visibility >>>> > > > > > > >>>> > > > > > > I will investigate further today. >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljoscha Krettek < >>>> > > aljos...@apache.org >>>> > > > > >>>> > > > > > > wrote: >>>> > > > > > > >>>> > > > > > >> On 06.07.20 20:39, Stephan Ewen wrote: >>>> > > > > > >> > - Did sink checkpoint notifications change in a >>>> relevant >>>> > way, >>>> > > > for >>>> > > > > > >> example >>>> > > > > > >> > due to some Kafka issues we addressed in 1.11 (@Aljoscha >>>> > maybe?) >>>> > > > > > >> >>>> > > > > > >> I think that's unrelated: the Kafka fixes were isolated in >>>> Kafka >>>> > > and >>>> > > > > the >>>> > > > > > >> one bug I discovered on the way was about the Task reaper. >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >> On 07.07.20 17:51, Zhijiang wrote: >>>> > > > > > >> > Sorry for my misunderstood of the previous information, >>>> > Thomas. >>>> > > I >>>> > > > > was >>>> > > > > > >> assuming that the sync checkpoint duration increased after >>>> > upgrade >>>> > > > as >>>> > > > > it >>>> > > > > > >> was mentioned before. >>>> > > > > > >> > >>>> > > > > > >> > If I remembered correctly, the memory state backend also >>>> has >>>> > the >>>> > > > > same >>>> > > > > > >> issue? If so, we can dismiss the rocksDB state changes. As >>>> the >>>> > > slot >>>> > > > > > sharing >>>> > > > > > >> enabled, the downstream and upstream should >>>> > > > > > >> > probably deployed into the same slot, then no network >>>> shuffle >>>> > > > > effect. >>>> > > > > > >> > >>>> > > > > > >> > I think we need to find out whether it has other symptoms >>>> > > changed >>>> > > > > > >> besides the performance regression to further figure out >>>> the >>>> > > scope. >>>> > > > > > >> > E.g. any metrics changes, the number of TaskManager and >>>> the >>>> > > number >>>> > > > > of >>>> > > > > > >> slots per TaskManager from deployment changes. >>>> > > > > > >> > 40% regression is really big, I guess the changes should >>>> also >>>> > be >>>> > > > > > >> reflected in other places. >>>> > > > > > >> > >>>> > > > > > >> > I am not sure whether we can reproduce the regression in >>>> our >>>> > AWS >>>> > > > > > >> environment by writing any Kinesis jobs, since there are >>>> also >>>> > > normal >>>> > > > > > >> Kinesis jobs as Thomas mentioned after upgrade. >>>> > > > > > >> > So it probably looks like to touch some corner case. I >>>> am very >>>> > > > > willing >>>> > > > > > >> to provide any help for debugging if possible. >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > Best, >>>> > > > > > >> > Zhijiang >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > ------------------------------------------------------------------ >>>> > > > > > >> > From:Thomas Weise <t...@apache.org> >>>> > > > > > >> > Send Time:2020年7月7日(星期二) 23:01 >>>> > > > > > >> > To:Stephan Ewen <se...@apache.org> >>>> > > > > > >> > Cc:Aljoscha Krettek <aljos...@apache.org>; Arvid Heise < >>>> > > > > > >> ar...@ververica.com>; Zhijiang <wangzhijiang...@aliyun.com >>>> > >>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] Release >>>> > > 1.11.0, >>>> > > > > > >> release candidate #4) >>>> > > > > > >> > >>>> > > > > > >> > We are deploying our apps with FlinkK8sOperator. We have >>>> one >>>> > job >>>> > > > > that >>>> > > > > > >> works as expected after the upgrade and the one discussed >>>> here >>>> > > that >>>> > > > > has >>>> > > > > > the >>>> > > > > > >> performance regression. >>>> > > > > > >> > >>>> > > > > > >> > "The performance regression is obvious caused by long >>>> duration >>>> > > of >>>> > > > > sync >>>> > > > > > >> checkpoint process in Kinesis sink operator, which would >>>> block >>>> > the >>>> > > > > > normal >>>> > > > > > >> data processing until back pressure the source." >>>> > > > > > >> > >>>> > > > > > >> > That's a constant. Before (1.10) and upgrade have the >>>> same >>>> > sync >>>> > > > > > >> checkpointing time. The question is what change came in >>>> with the >>>> > > > > > upgrade. >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > On Tue, Jul 7, 2020 at 7:33 AM Stephan Ewen < >>>> se...@apache.org >>>> > > >>>> > > > > wrote: >>>> > > > > > >> > >>>> > > > > > >> > @Thomas Just one thing real quick: Are you using the >>>> > standalone >>>> > > > > setup >>>> > > > > > >> scripts (like start-cluster.sh, and the former "slaves" >>>> file) ? >>>> > > > > > >> > Be aware that this is now called "workers" because of >>>> avoiding >>>> > > > > > >> sensitive names. >>>> > > > > > >> > In one internal benchmark we saw quite a lot of slowdown >>>> > > > initially, >>>> > > > > > >> before seeing that the cluster was not a distributed >>>> cluster any >>>> > > > more >>>> > > > > > ;-) >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > On Tue, Jul 7, 2020 at 9:08 AM Zhijiang < >>>> > > > wangzhijiang...@aliyun.com >>>> > > > > > >>>> > > > > > >> wrote: >>>> > > > > > >> > Thanks for this kickoff and help analysis, Stephan! >>>> > > > > > >> > Thanks for the further feedback and investigation, >>>> Thomas! >>>> > > > > > >> > >>>> > > > > > >> > The performance regression is obvious caused by long >>>> duration >>>> > of >>>> > > > > sync >>>> > > > > > >> checkpoint process in Kinesis sink operator, which would >>>> block >>>> > the >>>> > > > > > normal >>>> > > > > > >> data processing until back pressure the source. >>>> > > > > > >> > Maybe we could dig into the process of sync execution in >>>> > > > checkpoint. >>>> > > > > > >> E.g. break down the steps inside respective >>>> > operator#snapshotState >>>> > > > to >>>> > > > > > >> statistic which operation cost most of the time, then >>>> > > > > > >> > we might probably find the root cause to bring such cost. >>>> > > > > > >> > >>>> > > > > > >> > Look forward to the further progress. :) >>>> > > > > > >> > >>>> > > > > > >> > Best, >>>> > > > > > >> > Zhijiang >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > ------------------------------------------------------------------ >>>> > > > > > >> > From:Stephan Ewen <se...@apache.org> >>>> > > > > > >> > Send Time:2020年7月7日(星期二) 14:52 >>>> > > > > > >> > To:Thomas Weise <t...@apache.org> >>>> > > > > > >> > Cc:Stephan Ewen <se...@apache.org>; Zhijiang < >>>> > > > > > >> wangzhijiang...@aliyun.com>; Aljoscha Krettek < >>>> > > aljos...@apache.org >>>> > > > >; >>>> > > > > > >> Arvid Heise <ar...@ververica.com> >>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] Release >>>> > > 1.11.0, >>>> > > > > > >> release candidate #4) >>>> > > > > > >> > >>>> > > > > > >> > Thank you for the digging so deeply. >>>> > > > > > >> > Mysterious think this regression. >>>> > > > > > >> > >>>> > > > > > >> > On Mon, Jul 6, 2020, 22:56 Thomas Weise <t...@apache.org> >>>> > wrote: >>>> > > > > > >> > @Stephan: yes, I refer to sync time in the web UI (it is >>>> > > unchanged >>>> > > > > > >> between 1.10 and 1.11 for the specific pipeline). >>>> > > > > > >> > >>>> > > > > > >> > I verified that increasing the checkpointing interval >>>> does not >>>> > > > make >>>> > > > > a >>>> > > > > > >> difference. >>>> > > > > > >> > >>>> > > > > > >> > I looked at the Kinesis connector changes since 1.10.1 >>>> and >>>> > don't >>>> > > > see >>>> > > > > > >> anything that could cause this. >>>> > > > > > >> > >>>> > > > > > >> > Another pipeline that is using the Kinesis consumer (but >>>> not >>>> > the >>>> > > > > > >> producer) performs as expected. >>>> > > > > > >> > >>>> > > > > > >> > I tried reverting the AWS SDK version change, symptoms >>>> remain >>>> > > > > > unchanged: >>>> > > > > > >> > >>>> > > > > > >> > diff --git >>>> a/flink-connectors/flink-connector-kinesis/pom.xml >>>> > > > > > >> b/flink-connectors/flink-connector-kinesis/pom.xml >>>> > > > > > >> > index a6abce23ba..741743a05e 100644 >>>> > > > > > >> > --- a/flink-connectors/flink-connector-kinesis/pom.xml >>>> > > > > > >> > +++ b/flink-connectors/flink-connector-kinesis/pom.xml >>>> > > > > > >> > @@ -33,7 +33,7 @@ under the License. >>>> > > > > > >> > >>>> > > > > > >> >>>> > > > > >>>> > > >>>> <artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId> >>>> > > > > > >> > <name>flink-connector-kinesis</name> >>>> > > > > > >> > <properties> >>>> > > > > > >> > - >>>> <aws.sdk.version>1.11.754</aws.sdk.version> >>>> > > > > > >> > + >>>> <aws.sdk.version>1.11.603</aws.sdk.version> >>>> > > > > > >> > >>>> > > > > > >> <aws.kinesis-kcl.version>1.11.2</aws.kinesis-kcl.version> >>>> > > > > > >> > >>>> > > > > > >> <aws.kinesis-kpl.version>0.14.0</aws.kinesis-kpl.version> >>>> > > > > > >> > >>>> > > > > > >> >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> <aws.dynamodbstreams-kinesis-adapter.version>1.5.0</aws.dynamodbstreams-kinesis-adapter.version> >>>> > > > > > >> > >>>> > > > > > >> > I'm planning to take a look with a profiler next. >>>> > > > > > >> > >>>> > > > > > >> > Thomas >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > On Mon, Jul 6, 2020 at 11:40 AM Stephan Ewen < >>>> > se...@apache.org> >>>> > > > > > wrote: >>>> > > > > > >> > Hi all! >>>> > > > > > >> > >>>> > > > > > >> > Forking this thread out of the release vote thread. >>>> > > > > > >> > From what Thomas describes, it really sounds like a >>>> > > sink-specific >>>> > > > > > >> issue. >>>> > > > > > >> > >>>> > > > > > >> > @Thomas: When you say sink has a long synchronous >>>> checkpoint >>>> > > time, >>>> > > > > you >>>> > > > > > >> mean the time that is shown as "sync time" on the metrics >>>> and >>>> > web >>>> > > > UI? >>>> > > > > > That >>>> > > > > > >> is not including any network buffer related operations. It >>>> is >>>> > > purely >>>> > > > > the >>>> > > > > > >> operator's time. >>>> > > > > > >> > >>>> > > > > > >> > Can we dig into the changes we did in sinks: >>>> > > > > > >> > - Kinesis version upgrade, AWS library updates >>>> > > > > > >> > >>>> > > > > > >> > - Could it be that some call (checkpoint complete) >>>> that was >>>> > > > > > >> previously (1.10) in a separate thread is not in the >>>> mailbox and >>>> > > > this >>>> > > > > > >> simply reduces the number of threads that do the work? >>>> > > > > > >> > >>>> > > > > > >> > - Did sink checkpoint notifications change in a >>>> relevant >>>> > way, >>>> > > > for >>>> > > > > > >> example due to some Kafka issues we addressed in 1.11 >>>> (@Aljoscha >>>> > > > > maybe?) >>>> > > > > > >> > >>>> > > > > > >> > Best, >>>> > > > > > >> > Stephan >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > On Sun, Jul 5, 2020 at 7:10 AM Zhijiang < >>>> > > > wangzhijiang...@aliyun.com >>>> > > > > > .invalid> >>>> > > > > > >> wrote: >>>> > > > > > >> > Hi Thomas, >>>> > > > > > >> > >>>> > > > > > >> > Regarding [2], it has more detail infos in the Jira >>>> > > description >>>> > > > ( >>>> > > > > > >> https://issues.apache.org/jira/browse/FLINK-16404). >>>> > > > > > >> > >>>> > > > > > >> > I can also give some basic explanations here to >>>> dismiss the >>>> > > > > concern. >>>> > > > > > >> > 1. In the past, the following buffers after the >>>> barrier will >>>> > > be >>>> > > > > > >> cached on downstream side before alignment. >>>> > > > > > >> > 2. In 1.11, the upstream would not send the buffers >>>> after >>>> > the >>>> > > > > > >> barrier. When the downstream finishes the alignment, it >>>> will >>>> > > notify >>>> > > > > the >>>> > > > > > >> downstream of continuing sending following buffers, since >>>> it can >>>> > > > > process >>>> > > > > > >> them after alignment. >>>> > > > > > >> > 3. The only difference is that the temporary blocked >>>> buffers >>>> > > are >>>> > > > > > >> cached either on downstream side or on upstream side before >>>> > > > alignment. >>>> > > > > > >> > 4. The side effect would be the additional >>>> notification cost >>>> > > for >>>> > > > > > >> every barrier alignment. If the downstream and upstream are >>>> > > deployed >>>> > > > > in >>>> > > > > > >> separate TaskManager, the cost is network transport delay >>>> (the >>>> > > > effect >>>> > > > > > can >>>> > > > > > >> be ignored based on our testing with 1s checkpoint >>>> interval). >>>> > For >>>> > > > > > sharing >>>> > > > > > >> slot in your case, the cost is only one method call in >>>> > processor, >>>> > > > can >>>> > > > > be >>>> > > > > > >> ignored also. >>>> > > > > > >> > >>>> > > > > > >> > You mentioned "In this case, the downstream task has a >>>> high >>>> > > > > average >>>> > > > > > >> checkpoint duration(~30s, sync part)." This duration is not >>>> > > > reflecting >>>> > > > > > the >>>> > > > > > >> changes above, and it is only indicating the duration for >>>> > calling >>>> > > > > > >> `Operation.snapshotState`. >>>> > > > > > >> > If this duration is beyond your expectation, you can >>>> check >>>> > or >>>> > > > > debug >>>> > > > > > >> whether the source/sink operations might take more time to >>>> > finish >>>> > > > > > >> `snapshotState` in practice. E.g. you can >>>> > > > > > >> > make the implementation of this method as empty to >>>> further >>>> > > > verify >>>> > > > > > the >>>> > > > > > >> effect. >>>> > > > > > >> > >>>> > > > > > >> > Best, >>>> > > > > > >> > Zhijiang >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > >>>> ------------------------------------------------------------------ >>>> > > > > > >> > From:Thomas Weise <t...@apache.org> >>>> > > > > > >> > Send Time:2020年7月5日(星期日) 12:22 >>>> > > > > > >> > To:dev <dev@flink.apache.org>; Zhijiang < >>>> > > > > wangzhijiang...@aliyun.com >>>> > > > > > > >>>> > > > > > >> > Cc:Yingjie Cao <kevin.ying...@gmail.com> >>>> > > > > > >> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4 >>>> > > > > > >> > >>>> > > > > > >> > Hi Zhijiang, >>>> > > > > > >> > >>>> > > > > > >> > Could you please point me to more details regarding: >>>> "[2]: >>>> > > Delay >>>> > > > > > send >>>> > > > > > >> the >>>> > > > > > >> > following buffers after checkpoint barrier on upstream >>>> side >>>> > > > until >>>> > > > > > >> barrier >>>> > > > > > >> > alignment on downstream side." >>>> > > > > > >> > >>>> > > > > > >> > In this case, the downstream task has a high average >>>> > > checkpoint >>>> > > > > > >> duration >>>> > > > > > >> > (~30s, sync part). If there was a change to hold >>>> buffers >>>> > > > depending >>>> > > > > > on >>>> > > > > > >> > downstream performance, could this possibly apply to >>>> this >>>> > case >>>> > > > > (even >>>> > > > > > >> when >>>> > > > > > >> > there is no shuffle that would require alignment)? >>>> > > > > > >> > >>>> > > > > > >> > Thanks, >>>> > > > > > >> > Thomas >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > On Sat, Jul 4, 2020 at 7:39 AM Zhijiang < >>>> > > > > wangzhijiang...@aliyun.com >>>> > > > > > >> .invalid> >>>> > > > > > >> > wrote: >>>> > > > > > >> > >>>> > > > > > >> > > Hi Thomas, >>>> > > > > > >> > > >>>> > > > > > >> > > Thanks for the further update information. >>>> > > > > > >> > > >>>> > > > > > >> > > I guess we can dismiss the network stack changes, >>>> since in >>>> > > > your >>>> > > > > > >> case the >>>> > > > > > >> > > downstream and upstream would probably be deployed >>>> in the >>>> > > same >>>> > > > > > slot >>>> > > > > > >> > > bypassing the network data shuffle. >>>> > > > > > >> > > Also I guess release-1.11 will not bring general >>>> > performance >>>> > > > > > >> regression in >>>> > > > > > >> > > runtime engine, as we also did the performance >>>> testing for >>>> > > all >>>> > > > > > >> general >>>> > > > > > >> > > cases by [1] in real cluster before and the testing >>>> > results >>>> > > > > should >>>> > > > > > >> fit the >>>> > > > > > >> > > expectation. But we indeed did not test the specific >>>> > source >>>> > > > and >>>> > > > > > sink >>>> > > > > > >> > > connectors yet as I known. >>>> > > > > > >> > > >>>> > > > > > >> > > Regarding your performance regression with 40%, I >>>> wonder >>>> > it >>>> > > is >>>> > > > > > >> probably >>>> > > > > > >> > > related to specific source/sink changes (e.g. >>>> kinesis) or >>>> > > > > > >> environment >>>> > > > > > >> > > issues with corner case. >>>> > > > > > >> > > If possible, it would be helpful to further locate >>>> whether >>>> > > the >>>> > > > > > >> regression >>>> > > > > > >> > > is caused by kinesis, by replacing the kinesis >>>> source & >>>> > sink >>>> > > > and >>>> > > > > > >> keeping >>>> > > > > > >> > > the others same. >>>> > > > > > >> > > >>>> > > > > > >> > > As you said, it would be efficient to contact with >>>> you >>>> > > > directly >>>> > > > > > >> next week >>>> > > > > > >> > > to further discuss this issue. And we are >>>> willing/eager to >>>> > > > > provide >>>> > > > > > >> any help >>>> > > > > > >> > > to resolve this issue soon. >>>> > > > > > >> > > >>>> > > > > > >> > > Besides that, I guess this issue should not be the >>>> blocker >>>> > > for >>>> > > > > the >>>> > > > > > >> > > release, since it is probably a corner case based on >>>> the >>>> > > > current >>>> > > > > > >> analysis. >>>> > > > > > >> > > If we really conclude anything need to be resolved >>>> after >>>> > the >>>> > > > > final >>>> > > > > > >> > > release, then we can also make the next minor >>>> > release-1.11.1 >>>> > > > > come >>>> > > > > > >> soon. >>>> > > > > > >> > > >>>> > > > > > >> > > [1] >>>> https://issues.apache.org/jira/browse/FLINK-18433 >>>> > > > > > >> > > >>>> > > > > > >> > > Best, >>>> > > > > > >> > > Zhijiang >>>> > > > > > >> > > >>>> > > > > > >> > > >>>> > > > > > >> > > >>>> > > > > >>>> ------------------------------------------------------------------ >>>> > > > > > >> > > From:Thomas Weise <t...@apache.org> >>>> > > > > > >> > > Send Time:2020年7月4日(星期六) 12:26 >>>> > > > > > >> > > To:dev <dev@flink.apache.org>; Zhijiang < >>>> > > > > > wangzhijiang...@aliyun.com >>>> > > > > > >> > >>>> > > > > > >> > > Cc:Yingjie Cao <kevin.ying...@gmail.com> >>>> > > > > > >> > > Subject:Re: [VOTE] Release 1.11.0, release candidate >>>> #4 >>>> > > > > > >> > > >>>> > > > > > >> > > Hi Zhijiang, >>>> > > > > > >> > > >>>> > > > > > >> > > It will probably be best if we connect next week and >>>> > discuss >>>> > > > the >>>> > > > > > >> issue >>>> > > > > > >> > > directly since this could be quite difficult to >>>> reproduce. >>>> > > > > > >> > > >>>> > > > > > >> > > Before the testing result on our side comes out for >>>> your >>>> > > > > > respective >>>> > > > > > >> job >>>> > > > > > >> > > case, I have some other questions to confirm for >>>> further >>>> > > > > analysis: >>>> > > > > > >> > > - How much percentage regression you found after >>>> > > > switching >>>> > > > > to >>>> > > > > > >> 1.11? >>>> > > > > > >> > > >>>> > > > > > >> > > ~40% throughput decline >>>> > > > > > >> > > >>>> > > > > > >> > > - Are there any network bottleneck in your >>>> cluster? >>>> > > E.g. >>>> > > > > the >>>> > > > > > >> network >>>> > > > > > >> > > bandwidth is full caused by other jobs? If so, it >>>> might >>>> > have >>>> > > > > more >>>> > > > > > >> effects >>>> > > > > > >> > > by above [2] >>>> > > > > > >> > > >>>> > > > > > >> > > The test runs on a k8s cluster that is also used for >>>> other >>>> > > > > > >> production jobs. >>>> > > > > > >> > > There is no reason be believe network is the >>>> bottleneck. >>>> > > > > > >> > > >>>> > > > > > >> > > - Did you adjust the default network buffer >>>> setting? >>>> > > E.g. >>>> > > > > > >> > > >>>> "taskmanager.network.memory.floating-buffers-per-gate" or >>>> > > > > > >> > > "taskmanager.network.memory.buffers-per-channel" >>>> > > > > > >> > > >>>> > > > > > >> > > The job is using the defaults, i.e we don't >>>> configure the >>>> > > > > > settings. >>>> > > > > > >> If you >>>> > > > > > >> > > want me to try specific settings in the hope that it >>>> will >>>> > > help >>>> > > > > to >>>> > > > > > >> isolate >>>> > > > > > >> > > the issue please let me know. >>>> > > > > > >> > > >>>> > > > > > >> > > - I guess the topology has three vertexes >>>> > > > "KinesisConsumer >>>> > > > > -> >>>> > > > > > >> Chained >>>> > > > > > >> > > FlatMap -> KinesisProducer", and the partition mode >>>> for >>>> > > > > > >> "KinesisConsumer -> >>>> > > > > > >> > > FlatMap" and "FlatMap->KinesisProducer" are both >>>> > "forward"? >>>> > > If >>>> > > > > so, >>>> > > > > > >> the edge >>>> > > > > > >> > > connection is one-to-one, not all-to-all, then the >>>> above >>>> > > > [1][2] >>>> > > > > > >> should no >>>> > > > > > >> > > effects in theory with default network buffer >>>> setting. >>>> > > > > > >> > > >>>> > > > > > >> > > There are only 2 vertices and the edge is "forward". >>>> > > > > > >> > > >>>> > > > > > >> > > - By slot sharing, I guess these three vertex >>>> > > parallelism >>>> > > > > task >>>> > > > > > >> would >>>> > > > > > >> > > probably be deployed into the same slot, then the >>>> data >>>> > > shuffle >>>> > > > > is >>>> > > > > > >> by memory >>>> > > > > > >> > > queue, not network stack. If so, the above [2] >>>> should no >>>> > > > effect. >>>> > > > > > >> > > >>>> > > > > > >> > > Yes, vertices share slots. >>>> > > > > > >> > > >>>> > > > > > >> > > - I also saw some Jira changes for kinesis in >>>> this >>>> > > > release, >>>> > > > > > >> could you >>>> > > > > > >> > > confirm that these changes would not effect the >>>> > performance? >>>> > > > > > >> > > >>>> > > > > > >> > > I will need to take a look. 1.10 already had a >>>> regression >>>> > > > > > >> introduced by the >>>> > > > > > >> > > Kinesis producer update. >>>> > > > > > >> > > >>>> > > > > > >> > > >>>> > > > > > >> > > Thanks, >>>> > > > > > >> > > Thomas >>>> > > > > > >> > > >>>> > > > > > >> > > >>>> > > > > > >> > > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang < >>>> > > > > > >> wangzhijiang...@aliyun.com >>>> > > > > > >> > > .invalid> >>>> > > > > > >> > > wrote: >>>> > > > > > >> > > >>>> > > > > > >> > > > Hi Thomas, >>>> > > > > > >> > > > >>>> > > > > > >> > > > Thanks for your reply with rich information! >>>> > > > > > >> > > > >>>> > > > > > >> > > > We are trying to reproduce your case in our >>>> cluster to >>>> > > > further >>>> > > > > > >> verify it, >>>> > > > > > >> > > > and @Yingjie Cao is working on it now. >>>> > > > > > >> > > > As we have not kinesis consumer and producer >>>> > internally, >>>> > > so >>>> > > > > we >>>> > > > > > >> will >>>> > > > > > >> > > > construct the common source and sink instead in >>>> the case >>>> > > of >>>> > > > > > >> backpressure. >>>> > > > > > >> > > > >>>> > > > > > >> > > > Firstly, we can dismiss the rockdb factor in this >>>> > release, >>>> > > > > since >>>> > > > > > >> you also >>>> > > > > > >> > > > mentioned that "filesystem leads to same symptoms". >>>> > > > > > >> > > > >>>> > > > > > >> > > > Secondly, if my understanding is right, you >>>> emphasis >>>> > that >>>> > > > the >>>> > > > > > >> regression >>>> > > > > > >> > > > only exists for the jobs with low checkpoint >>>> interval >>>> > > (10s). >>>> > > > > > >> > > > Based on that, I have two suspicions with the >>>> network >>>> > > > related >>>> > > > > > >> changes in >>>> > > > > > >> > > > this release: >>>> > > > > > >> > > > - [1]: Limited the maximum backlog value >>>> (default >>>> > 10) >>>> > > in >>>> > > > > > >> subpartition >>>> > > > > > >> > > > queue. >>>> > > > > > >> > > > - [2]: Delay send the following buffers after >>>> > > checkpoint >>>> > > > > > >> barrier on >>>> > > > > > >> > > > upstream side until barrier alignment on downstream >>>> > side. >>>> > > > > > >> > > > >>>> > > > > > >> > > > These changes are motivated for reducing the >>>> in-flight >>>> > > > buffers >>>> > > > > > to >>>> > > > > > >> speedup >>>> > > > > > >> > > > checkpoint especially in the case of backpressure. >>>> > > > > > >> > > > In theory they should have very minor performance >>>> effect >>>> > > and >>>> > > > > > >> actually we >>>> > > > > > >> > > > also tested in cluster to verify within expectation >>>> > before >>>> > > > > > >> merging them, >>>> > > > > > >> > > > but maybe there are other corner cases we have not >>>> > > thought >>>> > > > of >>>> > > > > > >> before. >>>> > > > > > >> > > > >>>> > > > > > >> > > > Before the testing result on our side comes out >>>> for your >>>> > > > > > >> respective job >>>> > > > > > >> > > > case, I have some other questions to confirm for >>>> further >>>> > > > > > analysis: >>>> > > > > > >> > > > - How much percentage regression you found >>>> after >>>> > > > > switching >>>> > > > > > >> to 1.11? >>>> > > > > > >> > > > - Are there any network bottleneck in your >>>> cluster? >>>> > > > E.g. >>>> > > > > > the >>>> > > > > > >> network >>>> > > > > > >> > > > bandwidth is full caused by other jobs? If so, it >>>> might >>>> > > have >>>> > > > > > more >>>> > > > > > >> effects >>>> > > > > > >> > > > by above [2] >>>> > > > > > >> > > > - Did you adjust the default network buffer >>>> > setting? >>>> > > > E.g. >>>> > > > > > >> > > > >>>> "taskmanager.network.memory.floating-buffers-per-gate" >>>> > or >>>> > > > > > >> > > > "taskmanager.network.memory.buffers-per-channel" >>>> > > > > > >> > > > - I guess the topology has three vertexes >>>> > > > > "KinesisConsumer >>>> > > > > > -> >>>> > > > > > >> > > Chained >>>> > > > > > >> > > > FlatMap -> KinesisProducer", and the partition >>>> mode for >>>> > > > > > >> "KinesisConsumer >>>> > > > > > >> > > -> >>>> > > > > > >> > > > FlatMap" and "FlatMap->KinesisProducer" are both >>>> > > "forward"? >>>> > > > If >>>> > > > > > >> so, the >>>> > > > > > >> > > edge >>>> > > > > > >> > > > connection is one-to-one, not all-to-all, then the >>>> above >>>> > > > > [1][2] >>>> > > > > > >> should no >>>> > > > > > >> > > > effects in theory with default network buffer >>>> setting. >>>> > > > > > >> > > > - By slot sharing, I guess these three vertex >>>> > > > parallelism >>>> > > > > > >> task would >>>> > > > > > >> > > > probably be deployed into the same slot, then the >>>> data >>>> > > > shuffle >>>> > > > > > is >>>> > > > > > >> by >>>> > > > > > >> > > memory >>>> > > > > > >> > > > queue, not network stack. If so, the above [2] >>>> should no >>>> > > > > effect. >>>> > > > > > >> > > > - I also saw some Jira changes for kinesis in >>>> this >>>> > > > > release, >>>> > > > > > >> could you >>>> > > > > > >> > > > confirm that these changes would not effect the >>>> > > performance? >>>> > > > > > >> > > > >>>> > > > > > >> > > > Best, >>>> > > > > > >> > > > Zhijiang >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >>>> ------------------------------------------------------------------ >>>> > > > > > >> > > > From:Thomas Weise <t...@apache.org> >>>> > > > > > >> > > > Send Time:2020年7月3日(星期五) 01:07 >>>> > > > > > >> > > > To:dev <dev@flink.apache.org>; Zhijiang < >>>> > > > > > >> wangzhijiang...@aliyun.com> >>>> > > > > > >> > > > Subject:Re: [VOTE] Release 1.11.0, release >>>> candidate #4 >>>> > > > > > >> > > > >>>> > > > > > >> > > > Hi Zhijiang, >>>> > > > > > >> > > > >>>> > > > > > >> > > > The performance degradation manifests in >>>> backpressure >>>> > > which >>>> > > > > > leads >>>> > > > > > >> to >>>> > > > > > >> > > > growing backlog in the source. I switched a few >>>> times >>>> > > > between >>>> > > > > > >> 1.10 and >>>> > > > > > >> > > 1.11 >>>> > > > > > >> > > > and the behavior is consistent. >>>> > > > > > >> > > > >>>> > > > > > >> > > > The DAG is: >>>> > > > > > >> > > > >>>> > > > > > >> > > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) >>>> > > -------- >>>> > > > > > >> forward >>>> > > > > > >> > > > ---------> KinesisProducer >>>> > > > > > >> > > > >>>> > > > > > >> > > > Parallelism: 160 >>>> > > > > > >> > > > No shuffle/rebalance. >>>> > > > > > >> > > > >>>> > > > > > >> > > > Checkpointing config: >>>> > > > > > >> > > > >>>> > > > > > >> > > > Checkpointing Mode Exactly Once >>>> > > > > > >> > > > Interval 10s >>>> > > > > > >> > > > Timeout 10m 0s >>>> > > > > > >> > > > Minimum Pause Between Checkpoints 10s >>>> > > > > > >> > > > Maximum Concurrent Checkpoints 1 >>>> > > > > > >> > > > Persist Checkpoints Externally Enabled (delete on >>>> > > > > cancellation) >>>> > > > > > >> > > > >>>> > > > > > >> > > > State backend: rocksdb (filesystem leads to same >>>> > > symptoms) >>>> > > > > > >> > > > Checkpoint size is tiny (500KB) >>>> > > > > > >> > > > >>>> > > > > > >> > > > An interesting difference to another job that I had >>>> > > upgraded >>>> > > > > > >> successfully >>>> > > > > > >> > > > is the low checkpointing interval. >>>> > > > > > >> > > > >>>> > > > > > >> > > > Thanks, >>>> > > > > > >> > > > Thomas >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang < >>>> > > > > > >> wangzhijiang...@aliyun.com >>>> > > > > > >> > > > .invalid> >>>> > > > > > >> > > > wrote: >>>> > > > > > >> > > > >>>> > > > > > >> > > > > Hi Thomas, >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Thanks for the efficient feedback. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Regarding the suggestion of adding the release >>>> notes >>>> > > > > document, >>>> > > > > > >> I agree >>>> > > > > > >> > > > > with your point. Maybe we should adjust the vote >>>> > > template >>>> > > > > > >> accordingly >>>> > > > > > >> > > in >>>> > > > > > >> > > > > the respective wiki to guide the following >>>> release >>>> > > > > processes. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Regarding the performance regression, could you >>>> > provide >>>> > > > some >>>> > > > > > >> more >>>> > > > > > >> > > details >>>> > > > > > >> > > > > for our better measurement or reproducing on our >>>> > sides? >>>> > > > > > >> > > > > E.g. I guess the topology only includes two >>>> vertexes >>>> > > > source >>>> > > > > > and >>>> > > > > > >> sink? >>>> > > > > > >> > > > > What is the parallelism for every vertex? >>>> > > > > > >> > > > > The upstream shuffles data to the downstream via >>>> > > rebalance >>>> > > > > > >> partitioner >>>> > > > > > >> > > or >>>> > > > > > >> > > > > other? >>>> > > > > > >> > > > > The checkpoint mode is exactly-once with rocksDB >>>> state >>>> > > > > > backend? >>>> > > > > > >> > > > > The backpressure happened in this case? >>>> > > > > > >> > > > > How much percentage regression in this case? >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Best, >>>> > > > > > >> > > > > Zhijiang >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> >>>> > ------------------------------------------------------------------ >>>> > > > > > >> > > > > From:Thomas Weise <t...@apache.org> >>>> > > > > > >> > > > > Send Time:2020年7月2日(星期四) 09:54 >>>> > > > > > >> > > > > To:dev <dev@flink.apache.org> >>>> > > > > > >> > > > > Subject:Re: [VOTE] Release 1.11.0, release >>>> candidate >>>> > #4 >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Hi Till, >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Yes, we don't have the setting in >>>> flink-conf.yaml. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Generally, we carry forward the existing >>>> configuration >>>> > > and >>>> > > > > any >>>> > > > > > >> change >>>> > > > > > >> > > to >>>> > > > > > >> > > > > default configuration values would impact the >>>> upgrade. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Yes, since it is an incompatible change I would >>>> state >>>> > it >>>> > > > in >>>> > > > > > the >>>> > > > > > >> release >>>> > > > > > >> > > > > notes. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > Thanks, >>>> > > > > > >> > > > > Thomas >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > BTW I found a performance regression while >>>> trying to >>>> > > > upgrade >>>> > > > > > >> another >>>> > > > > > >> > > > > pipeline with this RC. It is a simple Kinesis to >>>> > Kinesis >>>> > > > > job. >>>> > > > > > >> Wasn't >>>> > > > > > >> > > able >>>> > > > > > >> > > > > to pin it down yet, symptoms include increased >>>> > > checkpoint >>>> > > > > > >> alignment >>>> > > > > > >> > > time. >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann < >>>> > > > > > >> trohrm...@apache.org> >>>> > > > > > >> > > > > wrote: >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > > Hi Thomas, >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > just to confirm: When starting the image in >>>> local >>>> > > mode, >>>> > > > > then >>>> > > > > > >> you >>>> > > > > > >> > > don't >>>> > > > > > >> > > > > have >>>> > > > > > >> > > > > > any of the JobManager memory configuration >>>> settings >>>> > > > > > >> configured in the >>>> > > > > > >> > > > > > effective flink-conf.yaml, right? Does this >>>> mean >>>> > that >>>> > > > you >>>> > > > > > have >>>> > > > > > >> > > > explicitly >>>> > > > > > >> > > > > > removed `jobmanager.heap.size: 1024m` from the >>>> > default >>>> > > > > > >> configuration? >>>> > > > > > >> > > > If >>>> > > > > > >> > > > > > this is the case, then I believe it was more >>>> of an >>>> > > > > > >> unintentional >>>> > > > > > >> > > > artifact >>>> > > > > > >> > > > > > that it worked before and it has been >>>> corrected now >>>> > so >>>> > > > > that >>>> > > > > > >> one needs >>>> > > > > > >> > > > to >>>> > > > > > >> > > > > > specify the memory of the JM process >>>> explicitly. Do >>>> > > you >>>> > > > > > think >>>> > > > > > >> it >>>> > > > > > >> > > would >>>> > > > > > >> > > > > help >>>> > > > > > >> > > > > > to explicitly state this in the release notes? >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > Cheers, >>>> > > > > > >> > > > > > Till >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise < >>>> > > > > t...@apache.org >>>> > > > > > > >>>> > > > > > >> wrote: >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > > Thanks for preparing another RC! >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > As mentioned in the previous RC thread, it >>>> would >>>> > be >>>> > > > > super >>>> > > > > > >> helpful >>>> > > > > > >> > > if >>>> > > > > > >> > > > > the >>>> > > > > > >> > > > > > > release notes that are part of the >>>> documentation >>>> > can >>>> > > > be >>>> > > > > > >> included >>>> > > > > > >> > > [1]. >>>> > > > > > >> > > > > > It's >>>> > > > > > >> > > > > > > a significant time-saver to have read those >>>> first. >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > I found one more non-backward compatible >>>> change >>>> > that >>>> > > > > would >>>> > > > > > >> be worth >>>> > > > > > >> > > > > > > addressing/mentioning: >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > It is now necessary to configure the >>>> jobmanager >>>> > heap >>>> > > > > size >>>> > > > > > in >>>> > > > > > >> > > > > > > flink-conf.yaml (with either >>>> jobmanager.heap.size >>>> > > > > > >> > > > > > > or jobmanager.memory.heap.size). Why would I >>>> not >>>> > > want >>>> > > > to >>>> > > > > > do >>>> > > > > > >> that >>>> > > > > > >> > > > > anyways? >>>> > > > > > >> > > > > > > Well, we set it dynamically for a cluster >>>> > deployment >>>> > > > via >>>> > > > > > the >>>> > > > > > >> > > > > > > flinkk8soperator, but the container image >>>> can also >>>> > > be >>>> > > > > used >>>> > > > > > >> for >>>> > > > > > >> > > > testing >>>> > > > > > >> > > > > > with >>>> > > > > > >> > > > > > > local mode (./bin/jobmanager.sh >>>> start-foreground >>>> > > > local). >>>> > > > > > >> That will >>>> > > > > > >> > > > fail >>>> > > > > > >> > > > > > if >>>> > > > > > >> > > > > > > the heap wasn't configured and that's how I >>>> > noticed >>>> > > > it. >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > Thanks, >>>> > > > > > >> > > > > > > Thomas >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > [1] >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > >>>> > > > > > >> >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang < >>>> > > > > > >> > > wangzhijiang...@aliyun.com >>>> > > > > > >> > > > > > > .invalid> >>>> > > > > > >> > > > > > > wrote: >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > > > Hi everyone, >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > Please review and vote on the release >>>> candidate >>>> > #4 >>>> > > > for >>>> > > > > > the >>>> > > > > > >> > > version >>>> > > > > > >> > > > > > > 1.11.0, >>>> > > > > > >> > > > > > > > as follows: >>>> > > > > > >> > > > > > > > [ ] +1, Approve the release >>>> > > > > > >> > > > > > > > [ ] -1, Do not approve the release (please >>>> > provide >>>> > > > > > >> specific >>>> > > > > > >> > > > comments) >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > The complete staging area is available for >>>> your >>>> > > > > review, >>>> > > > > > >> which >>>> > > > > > >> > > > > includes: >>>> > > > > > >> > > > > > > > * JIRA release notes [1], >>>> > > > > > >> > > > > > > > * the official Apache source release and >>>> binary >>>> > > > > > >> convenience >>>> > > > > > >> > > > releases >>>> > > > > > >> > > > > to >>>> > > > > > >> > > > > > > be >>>> > > > > > >> > > > > > > > deployed to dist.apache.org [2], which are >>>> > signed >>>> > > > > with >>>> > > > > > >> the key >>>> > > > > > >> > > > with >>>> > > > > > >> > > > > > > > fingerprint >>>> > > 2DA85B93244FDFA19A6244500653C0A2CEA00D0E >>>> > > > > > [3], >>>> > > > > > >> > > > > > > > * all artifacts to be deployed to the Maven >>>> > > Central >>>> > > > > > >> Repository >>>> > > > > > >> > > [4], >>>> > > > > > >> > > > > > > > * source code tag "release-1.11.0-rc4" [5], >>>> > > > > > >> > > > > > > > * website pull request listing the new >>>> release >>>> > and >>>> > > > > > adding >>>> > > > > > >> > > > > announcement >>>> > > > > > >> > > > > > > > blog post [6]. >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > The vote will be open for at least 72 >>>> hours. It >>>> > is >>>> > > > > > >> adopted by >>>> > > > > > >> > > > > majority >>>> > > > > > >> > > > > > > > approval, with at least 3 PMC affirmative >>>> votes. >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > Thanks, >>>> > > > > > >> > > > > > > > Release Manager >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > [1] >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > >>>> > > > > > >> >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364 >>>> > > > > > >> > > > > > > > [2] >>>> > > > > > >> > > >>>> > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/ >>>> > > > > > >> > > > > > > > [3] >>>> > > > > > https://dist.apache.org/repos/dist/release/flink/KEYS >>>> > > > > > >> > > > > > > > [4] >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > >>>> > > > > > >> >>>> > > > > >>>> > > >>>> https://repository.apache.org/content/repositories/orgapacheflink-1377/ >>>> > > > > > >> > > > > > > > [5] >>>> > > > > > >> > > > >>>> > > > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4 >>>> > > > > > >> > > > > > > > [6] >>>> > https://github.com/apache/flink-web/pull/352 >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > > >>>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > >>>> > > > > > >> > > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> > >>>> > -- >>>> > Regards, >>>> > Roman >>>> > >>>> >>> >>> >>> -- >>> Regards, >>> Roman >>> >> -- Regards, Roman
