Thanks Thomas for reporting the problem, analysing which commit has caused and now for the verification that it was fixed :) Much appreciated.
Piotrek czw., 13 sie 2020 o 18:18 Thomas Weise <t...@apache.org> napisał(a): > Hi Roman, > > Thanks for working on this! I deployed the change and it appears to be > working as expected. > > Will monitor over a period of time to compare the checkpoint counts and get > back to you if there are still issues. > > Thomas > > > On Thu, Aug 13, 2020 at 3:41 AM Roman Khachatryan <ro...@data-artisans.com > > > wrote: > > > Hi Thomas, > > > > The fix is now merged to master and to release-1.11. > > So if you'd like you can check if it solves your problem (it would be > > helpful for us too). > > > > On Sat, Aug 8, 2020 at 9:26 AM Roman Khachatryan < > ro...@data-artisans.com> > > wrote: > > > >> Hi Thomas, > >> > >> Thanks a lot for the detailed information. > >> > >> I think the problem is in CheckpointCoordinator. It stores the last > >> checkpoint completion time after checking queued requests. > >> I've created a ticket to fix this: > >> https://issues.apache.org/jira/browse/FLINK-18856 > >> > >> > >> On Sat, Aug 8, 2020 at 5:25 AM Thomas Weise <t...@apache.org> wrote: > >> > >>> Just another update: > >>> > >>> The duration of snapshotState is capped by the Kinesis > >>> producer's "RecordTtl" setting (default 30s). The sleep time in > flushSync > >>> does not contribute to the observed behavior. > >>> > >>> I guess the open question is why, with the same settings, is 1.11 since > >>> commit 355184d69a8519d29937725c8d85e8465d7e3a90 processing more > checkpoints? > >>> > >>> > >>> On Fri, Aug 7, 2020 at 9:15 AM Thomas Weise <t...@apache.org> wrote: > >>> > >>>> Hi Roman, > >>>> > >>>> Here are the checkpoint summaries for both commits: > >>>> > >>>> > >>>> > https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit#slide=id.g86d15b2fc7_0_0 > >>>> > >>>> The config: > >>>> > >>>> CheckpointConfig checkpointConfig = env.getCheckpointConfig(); > >>>> > >>>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > >>>> checkpointConfig.setCheckpointInterval(*10_000*); > >>>> checkpointConfig.setMinPauseBetweenCheckpoints(*10_000*); > >>>> > >>>> > checkpointConfig.enableExternalizedCheckpoints(DELETE_ON_CANCELLATION); > >>>> checkpointConfig.setCheckpointTimeout(600_000); > >>>> checkpointConfig.setMaxConcurrentCheckpoints(1); > >>>> checkpointConfig.setFailOnCheckpointingErrors(true); > >>>> > >>>> The values marked bold when changed to *60_000* make the symptom > >>>> disappear. I meanwhile also verified that with the 1.11.0 release > commit. > >>>> > >>>> I will take a look at the sleep time issue. > >>>> > >>>> Thanks, > >>>> Thomas > >>>> > >>>> > >>>> On Fri, Aug 7, 2020 at 1:44 AM Roman Khachatryan < > >>>> ro...@data-artisans.com> wrote: > >>>> > >>>>> Hi Thomas, > >>>>> > >>>>> Thanks for your reply! > >>>>> > >>>>> I think you are right, we can remove this sleep and improve > >>>>> KinesisProducer. > >>>>> Probably, it's snapshotState can also be sped up by forcing records > >>>>> flush more often. > >>>>> Do you see that 30s checkpointing duration is caused > >>>>> by KinesisProducer (or maybe other operators)? > >>>>> > >>>>> I'd also like to understand the reason behind this increase in > >>>>> checkpoint frequency. > >>>>> Can you please share these values: > >>>>> - execution.checkpointing.min-pause > >>>>> - execution.checkpointing.max-concurrent-checkpoints > >>>>> - execution.checkpointing.timeout > >>>>> > >>>>> And what is the "new" observed checkpoint frequency (or how many > >>>>> checkpoints are created) compared to older versions? > >>>>> > >>>>> > >>>>> On Fri, Aug 7, 2020 at 4:49 AM Thomas Weise <t...@apache.org> wrote: > >>>>> > >>>>>> Hi Roman, > >>>>>> > >>>>>> Indeed there are more frequent checkpoints with this change! The > >>>>>> application was configured to checkpoint every 10s. With 1.10 ("good > >>>>>> commit"), that leads to fewer completed checkpoints compared to 1.11 > >>>>>> ("bad > >>>>>> commit"). Just to be clear, the only difference between the two runs > >>>>>> was > >>>>>> the commit 355184d69a8519d29937725c8d85e8465d7e3a90 > >>>>>> > >>>>>> Since the sync part of checkpoints with the Kinesis producer always > >>>>>> takes > >>>>>> ~30 seconds, the 10s configured checkpoint frequency really had no > >>>>>> effect > >>>>>> before 1.11. I confirmed that both commits perform comparably by > >>>>>> setting > >>>>>> the checkpoint frequency and min pause to 60s. > >>>>>> > >>>>>> I still have to verify with the final 1.11.0 release commit. > >>>>>> > >>>>>> It's probably good to take a look at the Kinesis producer. Is it > >>>>>> really > >>>>>> necessary to have 500ms sleep time? What's responsible for the ~30s > >>>>>> duration in snapshotState? > >>>>>> > >>>>>> As things stand it doesn't make sense to use checkpoint intervals < > >>>>>> 30s > >>>>>> when using the Kinesis producer. > >>>>>> > >>>>>> Thanks, > >>>>>> Thomas > >>>>>> > >>>>>> On Sat, Aug 1, 2020 at 2:53 PM Roman Khachatryan < > >>>>>> ro...@data-artisans.com> > >>>>>> wrote: > >>>>>> > >>>>>> > Hi Thomas, > >>>>>> > > >>>>>> > Thanks a lot for the analysis. > >>>>>> > > >>>>>> > The first thing that I'd check is whether checkpoints became more > >>>>>> frequent > >>>>>> > with this commit (as each of them adds at least 500ms if there is > >>>>>> at least > >>>>>> > one not sent record, according to > >>>>>> FlinkKinesisProducer.snapshotState). > >>>>>> > > >>>>>> > Can you share checkpointing statistics (1.10 vs 1.11 or last > "good" > >>>>>> vs > >>>>>> > first "bad" commits)? > >>>>>> > > >>>>>> > On Fri, Jul 31, 2020 at 5:29 AM Thomas Weise < > >>>>>> thomas.we...@gmail.com> > >>>>>> > wrote: > >>>>>> > > >>>>>> > > I run git bisect and the first commit that shows the regression > >>>>>> is: > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > >>>>>> > https://github.com/apache/flink/commit/355184d69a8519d29937725c8d85e8465d7e3a90 > >>>>>> > > > >>>>>> > > > >>>>>> > > On Thu, Jul 23, 2020 at 6:46 PM Kurt Young <ykt...@gmail.com> > >>>>>> wrote: > >>>>>> > > > >>>>>> > > > From my experience, java profilers are sometimes not accurate > >>>>>> enough to > >>>>>> > > > find out the performance regression > >>>>>> > > > root cause. In this case, I would suggest you try out intel > >>>>>> vtune > >>>>>> > > amplifier > >>>>>> > > > to watch more detailed metrics. > >>>>>> > > > > >>>>>> > > > Best, > >>>>>> > > > Kurt > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > On Fri, Jul 24, 2020 at 8:51 AM Thomas Weise <t...@apache.org> > >>>>>> wrote: > >>>>>> > > > > >>>>>> > > > > The cause of the issue is all but clear. > >>>>>> > > > > > >>>>>> > > > > Previously I had mentioned that there is no suspect change > to > >>>>>> the > >>>>>> > > Kinesis > >>>>>> > > > > connector and that I had reverted the AWS SDK change to no > >>>>>> effect. > >>>>>> > > > > > >>>>>> > > > > https://issues.apache.org/jira/browse/FLINK-17496 actually > >>>>>> fixed > >>>>>> > > another > >>>>>> > > > > regression in the previous release and is present before and > >>>>>> after. > >>>>>> > > > > > >>>>>> > > > > I repeated the run with 1.11.0 core and downgraded the > entire > >>>>>> Kinesis > >>>>>> > > > > connector to 1.10.1: Nothing changes, i.e. the regression is > >>>>>> still > >>>>>> > > > present. > >>>>>> > > > > Therefore we will need to look elsewhere for the root cause. > >>>>>> > > > > > >>>>>> > > > > Regarding the time spent in snapshotState, repeat runs > reveal > >>>>>> a wide > >>>>>> > > > range > >>>>>> > > > > for both versions, 1.10 and 1.11. So again this is nothing > >>>>>> pointing > >>>>>> > to > >>>>>> > > a > >>>>>> > > > > root cause. > >>>>>> > > > > > >>>>>> > > > > At this point, I have no ideas remaining other than doing a > >>>>>> bisect to > >>>>>> > > > find > >>>>>> > > > > the culprit. Any other suggestions? > >>>>>> > > > > > >>>>>> > > > > Thomas > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > On Thu, Jul 16, 2020 at 9:19 PM Zhijiang < > >>>>>> wangzhijiang...@aliyun.com > >>>>>> > > > > .invalid> > >>>>>> > > > > wrote: > >>>>>> > > > > > >>>>>> > > > > > Hi Thomas, > >>>>>> > > > > > > >>>>>> > > > > > Thanks for your further profiling information and glad to > >>>>>> see we > >>>>>> > > > already > >>>>>> > > > > > finalized the location to cause the regression. > >>>>>> > > > > > Actually I was also suspicious of the point of > >>>>>> #snapshotState in > >>>>>> > > > previous > >>>>>> > > > > > discussions since it indeed cost much time to block normal > >>>>>> operator > >>>>>> > > > > > processing. > >>>>>> > > > > > > >>>>>> > > > > > Based on your below feedback, the sleep time during > >>>>>> #snapshotState > >>>>>> > > > might > >>>>>> > > > > > be the main concern, and I also digged into the > >>>>>> implementation of > >>>>>> > > > > > FlinkKinesisProducer#snapshotState. > >>>>>> > > > > > while (producer.getOutstandingRecordsCount() > 0) { > >>>>>> > > > > > producer.flush(); > >>>>>> > > > > > try { > >>>>>> > > > > > Thread.sleep(500); > >>>>>> > > > > > } catch (InterruptedException e) { > >>>>>> > > > > > LOG.warn("Flushing was interrupted."); > >>>>>> > > > > > break; > >>>>>> > > > > > } > >>>>>> > > > > > } > >>>>>> > > > > > It seems that the sleep time is mainly affected by the > >>>>>> internal > >>>>>> > > > > operations > >>>>>> > > > > > inside KinesisProducer implementation provided by > >>>>>> amazonaws, which > >>>>>> > I > >>>>>> > > am > >>>>>> > > > > not > >>>>>> > > > > > quite familiar with. > >>>>>> > > > > > But I noticed there were two upgrades related to it in > >>>>>> > > release-1.11.0. > >>>>>> > > > > One > >>>>>> > > > > > is for upgrading amazon-kinesis-producer to 0.14.0 [1] and > >>>>>> another > >>>>>> > is > >>>>>> > > > for > >>>>>> > > > > > upgrading aws-sdk-version to 1.11.754 [2]. > >>>>>> > > > > > You mentioned that you already reverted the SDK upgrade to > >>>>>> verify > >>>>>> > no > >>>>>> > > > > > changes. Did you also revert the [1] to verify? > >>>>>> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-17496 > >>>>>> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-14881 > >>>>>> > > > > > > >>>>>> > > > > > Best, > >>>>>> > > > > > Zhijiang > >>>>>> > > > > > > >>>>>> ------------------------------------------------------------------ > >>>>>> > > > > > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > Send Time:2020年7月17日(星期五) 05:29 > >>>>>> > > > > > To:dev <dev@flink.apache.org> > >>>>>> > > > > > Cc:Zhijiang <wangzhijiang...@aliyun.com>; Stephan Ewen < > >>>>>> > > > se...@apache.org > >>>>>> > > > > >; > >>>>>> > > > > > Arvid Heise <ar...@ververica.com>; Aljoscha Krettek < > >>>>>> > > > aljos...@apache.org > >>>>>> > > > > > > >>>>>> > > > > > Subject:Re: Kinesis Performance Issue (was [VOTE] Release > >>>>>> 1.11.0, > >>>>>> > > > release > >>>>>> > > > > > candidate #4) > >>>>>> > > > > > > >>>>>> > > > > > Sorry for the delay. > >>>>>> > > > > > > >>>>>> > > > > > I confirmed that the regression is due to the sink > >>>>>> (unsurprising, > >>>>>> > > since > >>>>>> > > > > > another job with the same consumer, but not the producer, > >>>>>> runs as > >>>>>> > > > > > expected). > >>>>>> > > > > > > >>>>>> > > > > > As promised I did CPU profiling on the problematic > >>>>>> application, > >>>>>> > which > >>>>>> > > > > gives > >>>>>> > > > > > more insight into the regression [1] > >>>>>> > > > > > > >>>>>> > > > > > The screenshots show that the average time for > snapshotState > >>>>>> > > increases > >>>>>> > > > > from > >>>>>> > > > > > ~9s to ~28s. The data also shows the increase in sleep > time > >>>>>> during > >>>>>> > > > > > snapshotState. > >>>>>> > > > > > > >>>>>> > > > > > Does anyone, based on changes made in 1.11, have a theory > >>>>>> why? > >>>>>> > > > > > > >>>>>> > > > > > I had previously looked at the changes to the Kinesis > >>>>>> connector and > >>>>>> > > > also > >>>>>> > > > > > reverted the SDK upgrade, which did not change the > >>>>>> situation. > >>>>>> > > > > > > >>>>>> > > > > > It will likely be necessary to drill into the sink / > >>>>>> checkpointing > >>>>>> > > > > details > >>>>>> > > > > > to understand the cause of the problem. > >>>>>> > > > > > > >>>>>> > > > > > Let me know if anyone has specific questions that I can > >>>>>> answer from > >>>>>> > > the > >>>>>> > > > > > profiling results. > >>>>>> > > > > > > >>>>>> > > > > > Thomas > >>>>>> > > > > > > >>>>>> > > > > > [1] > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit?usp=sharing > >>>>>> > > > > > > >>>>>> > > > > > On Mon, Jul 13, 2020 at 11:14 AM Thomas Weise < > >>>>>> t...@apache.org> > >>>>>> > > wrote: > >>>>>> > > > > > > >>>>>> > > > > > > + dev@ for visibility > >>>>>> > > > > > > > >>>>>> > > > > > > I will investigate further today. > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljoscha Krettek < > >>>>>> > > aljos...@apache.org > >>>>>> > > > > > >>>>>> > > > > > > wrote: > >>>>>> > > > > > > > >>>>>> > > > > > >> On 06.07.20 20:39, Stephan Ewen wrote: > >>>>>> > > > > > >> > - Did sink checkpoint notifications change in a > >>>>>> relevant > >>>>>> > way, > >>>>>> > > > for > >>>>>> > > > > > >> example > >>>>>> > > > > > >> > due to some Kafka issues we addressed in 1.11 > >>>>>> (@Aljoscha > >>>>>> > maybe?) > >>>>>> > > > > > >> > >>>>>> > > > > > >> I think that's unrelated: the Kafka fixes were isolated > >>>>>> in Kafka > >>>>>> > > and > >>>>>> > > > > the > >>>>>> > > > > > >> one bug I discovered on the way was about the Task > >>>>>> reaper. > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > >> On 07.07.20 17:51, Zhijiang wrote: > >>>>>> > > > > > >> > Sorry for my misunderstood of the previous > information, > >>>>>> > Thomas. > >>>>>> > > I > >>>>>> > > > > was > >>>>>> > > > > > >> assuming that the sync checkpoint duration increased > >>>>>> after > >>>>>> > upgrade > >>>>>> > > > as > >>>>>> > > > > it > >>>>>> > > > > > >> was mentioned before. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > If I remembered correctly, the memory state backend > >>>>>> also has > >>>>>> > the > >>>>>> > > > > same > >>>>>> > > > > > >> issue? If so, we can dismiss the rocksDB state changes. > >>>>>> As the > >>>>>> > > slot > >>>>>> > > > > > sharing > >>>>>> > > > > > >> enabled, the downstream and upstream should > >>>>>> > > > > > >> > probably deployed into the same slot, then no network > >>>>>> shuffle > >>>>>> > > > > effect. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I think we need to find out whether it has other > >>>>>> symptoms > >>>>>> > > changed > >>>>>> > > > > > >> besides the performance regression to further figure > out > >>>>>> the > >>>>>> > > scope. > >>>>>> > > > > > >> > E.g. any metrics changes, the number of TaskManager > >>>>>> and the > >>>>>> > > number > >>>>>> > > > > of > >>>>>> > > > > > >> slots per TaskManager from deployment changes. > >>>>>> > > > > > >> > 40% regression is really big, I guess the changes > >>>>>> should also > >>>>>> > be > >>>>>> > > > > > >> reflected in other places. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I am not sure whether we can reproduce the regression > >>>>>> in our > >>>>>> > AWS > >>>>>> > > > > > >> environment by writing any Kinesis jobs, since there > are > >>>>>> also > >>>>>> > > normal > >>>>>> > > > > > >> Kinesis jobs as Thomas mentioned after upgrade. > >>>>>> > > > > > >> > So it probably looks like to touch some corner case. > I > >>>>>> am very > >>>>>> > > > > willing > >>>>>> > > > > > >> to provide any help for debugging if possible. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Best, > >>>>>> > > > > > >> > Zhijiang > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > ------------------------------------------------------------------ > >>>>>> > > > > > >> > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > Send Time:2020年7月7日(星期二) 23:01 > >>>>>> > > > > > >> > To:Stephan Ewen <se...@apache.org> > >>>>>> > > > > > >> > Cc:Aljoscha Krettek <aljos...@apache.org>; Arvid > >>>>>> Heise < > >>>>>> > > > > > >> ar...@ververica.com>; Zhijiang < > >>>>>> wangzhijiang...@aliyun.com> > >>>>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] > >>>>>> Release > >>>>>> > > 1.11.0, > >>>>>> > > > > > >> release candidate #4) > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > We are deploying our apps with FlinkK8sOperator. We > >>>>>> have one > >>>>>> > job > >>>>>> > > > > that > >>>>>> > > > > > >> works as expected after the upgrade and the one > >>>>>> discussed here > >>>>>> > > that > >>>>>> > > > > has > >>>>>> > > > > > the > >>>>>> > > > > > >> performance regression. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > "The performance regression is obvious caused by long > >>>>>> duration > >>>>>> > > of > >>>>>> > > > > sync > >>>>>> > > > > > >> checkpoint process in Kinesis sink operator, which > would > >>>>>> block > >>>>>> > the > >>>>>> > > > > > normal > >>>>>> > > > > > >> data processing until back pressure the source." > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > That's a constant. Before (1.10) and upgrade have the > >>>>>> same > >>>>>> > sync > >>>>>> > > > > > >> checkpointing time. The question is what change came in > >>>>>> with the > >>>>>> > > > > > upgrade. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Tue, Jul 7, 2020 at 7:33 AM Stephan Ewen < > >>>>>> se...@apache.org > >>>>>> > > > >>>>>> > > > > wrote: > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > @Thomas Just one thing real quick: Are you using the > >>>>>> > standalone > >>>>>> > > > > setup > >>>>>> > > > > > >> scripts (like start-cluster.sh, and the former "slaves" > >>>>>> file) ? > >>>>>> > > > > > >> > Be aware that this is now called "workers" because of > >>>>>> avoiding > >>>>>> > > > > > >> sensitive names. > >>>>>> > > > > > >> > In one internal benchmark we saw quite a lot of > >>>>>> slowdown > >>>>>> > > > initially, > >>>>>> > > > > > >> before seeing that the cluster was not a distributed > >>>>>> cluster any > >>>>>> > > > more > >>>>>> > > > > > ;-) > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Tue, Jul 7, 2020 at 9:08 AM Zhijiang < > >>>>>> > > > wangzhijiang...@aliyun.com > >>>>>> > > > > > > >>>>>> > > > > > >> wrote: > >>>>>> > > > > > >> > Thanks for this kickoff and help analysis, Stephan! > >>>>>> > > > > > >> > Thanks for the further feedback and investigation, > >>>>>> Thomas! > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > The performance regression is obvious caused by long > >>>>>> duration > >>>>>> > of > >>>>>> > > > > sync > >>>>>> > > > > > >> checkpoint process in Kinesis sink operator, which > would > >>>>>> block > >>>>>> > the > >>>>>> > > > > > normal > >>>>>> > > > > > >> data processing until back pressure the source. > >>>>>> > > > > > >> > Maybe we could dig into the process of sync execution > >>>>>> in > >>>>>> > > > checkpoint. > >>>>>> > > > > > >> E.g. break down the steps inside respective > >>>>>> > operator#snapshotState > >>>>>> > > > to > >>>>>> > > > > > >> statistic which operation cost most of the time, then > >>>>>> > > > > > >> > we might probably find the root cause to bring such > >>>>>> cost. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Look forward to the further progress. :) > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Best, > >>>>>> > > > > > >> > Zhijiang > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > ------------------------------------------------------------------ > >>>>>> > > > > > >> > From:Stephan Ewen <se...@apache.org> > >>>>>> > > > > > >> > Send Time:2020年7月7日(星期二) 14:52 > >>>>>> > > > > > >> > To:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > Cc:Stephan Ewen <se...@apache.org>; Zhijiang < > >>>>>> > > > > > >> wangzhijiang...@aliyun.com>; Aljoscha Krettek < > >>>>>> > > aljos...@apache.org > >>>>>> > > > >; > >>>>>> > > > > > >> Arvid Heise <ar...@ververica.com> > >>>>>> > > > > > >> > Subject:Re: Kinesis Performance Issue (was [VOTE] > >>>>>> Release > >>>>>> > > 1.11.0, > >>>>>> > > > > > >> release candidate #4) > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Thank you for the digging so deeply. > >>>>>> > > > > > >> > Mysterious think this regression. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Mon, Jul 6, 2020, 22:56 Thomas Weise < > >>>>>> t...@apache.org> > >>>>>> > wrote: > >>>>>> > > > > > >> > @Stephan: yes, I refer to sync time in the web UI (it > >>>>>> is > >>>>>> > > unchanged > >>>>>> > > > > > >> between 1.10 and 1.11 for the specific pipeline). > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I verified that increasing the checkpointing interval > >>>>>> does not > >>>>>> > > > make > >>>>>> > > > > a > >>>>>> > > > > > >> difference. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I looked at the Kinesis connector changes since > 1.10.1 > >>>>>> and > >>>>>> > don't > >>>>>> > > > see > >>>>>> > > > > > >> anything that could cause this. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Another pipeline that is using the Kinesis consumer > >>>>>> (but not > >>>>>> > the > >>>>>> > > > > > >> producer) performs as expected. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I tried reverting the AWS SDK version change, > symptoms > >>>>>> remain > >>>>>> > > > > > unchanged: > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > diff --git > >>>>>> a/flink-connectors/flink-connector-kinesis/pom.xml > >>>>>> > > > > > >> b/flink-connectors/flink-connector-kinesis/pom.xml > >>>>>> > > > > > >> > index a6abce23ba..741743a05e 100644 > >>>>>> > > > > > >> > --- > a/flink-connectors/flink-connector-kinesis/pom.xml > >>>>>> > > > > > >> > +++ > b/flink-connectors/flink-connector-kinesis/pom.xml > >>>>>> > > > > > >> > @@ -33,7 +33,7 @@ under the License. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > >>>>>> > > > > > >>>>>> > > > >>>>>> > <artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId> > >>>>>> > > > > > >> > <name>flink-connector-kinesis</name> > >>>>>> > > > > > >> > <properties> > >>>>>> > > > > > >> > - > >>>>>> <aws.sdk.version>1.11.754</aws.sdk.version> > >>>>>> > > > > > >> > + > >>>>>> <aws.sdk.version>1.11.603</aws.sdk.version> > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > <aws.kinesis-kcl.version>1.11.2</aws.kinesis-kcl.version> > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > <aws.kinesis-kpl.version>0.14.0</aws.kinesis-kpl.version> > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > <aws.dynamodbstreams-kinesis-adapter.version>1.5.0</aws.dynamodbstreams-kinesis-adapter.version> > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I'm planning to take a look with a profiler next. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Thomas > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Mon, Jul 6, 2020 at 11:40 AM Stephan Ewen < > >>>>>> > se...@apache.org> > >>>>>> > > > > > wrote: > >>>>>> > > > > > >> > Hi all! > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Forking this thread out of the release vote thread. > >>>>>> > > > > > >> > From what Thomas describes, it really sounds like a > >>>>>> > > sink-specific > >>>>>> > > > > > >> issue. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > @Thomas: When you say sink has a long synchronous > >>>>>> checkpoint > >>>>>> > > time, > >>>>>> > > > > you > >>>>>> > > > > > >> mean the time that is shown as "sync time" on the > >>>>>> metrics and > >>>>>> > web > >>>>>> > > > UI? > >>>>>> > > > > > That > >>>>>> > > > > > >> is not including any network buffer related operations. > >>>>>> It is > >>>>>> > > purely > >>>>>> > > > > the > >>>>>> > > > > > >> operator's time. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Can we dig into the changes we did in sinks: > >>>>>> > > > > > >> > - Kinesis version upgrade, AWS library updates > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > - Could it be that some call (checkpoint complete) > >>>>>> that was > >>>>>> > > > > > >> previously (1.10) in a separate thread is not in the > >>>>>> mailbox and > >>>>>> > > > this > >>>>>> > > > > > >> simply reduces the number of threads that do the work? > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > - Did sink checkpoint notifications change in a > >>>>>> relevant > >>>>>> > way, > >>>>>> > > > for > >>>>>> > > > > > >> example due to some Kafka issues we addressed in 1.11 > >>>>>> (@Aljoscha > >>>>>> > > > > maybe?) > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Best, > >>>>>> > > > > > >> > Stephan > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Sun, Jul 5, 2020 at 7:10 AM Zhijiang < > >>>>>> > > > wangzhijiang...@aliyun.com > >>>>>> > > > > > .invalid> > >>>>>> > > > > > >> wrote: > >>>>>> > > > > > >> > Hi Thomas, > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Regarding [2], it has more detail infos in the Jira > >>>>>> > > description > >>>>>> > > > ( > >>>>>> > > > > > >> https://issues.apache.org/jira/browse/FLINK-16404). > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > I can also give some basic explanations here to > >>>>>> dismiss the > >>>>>> > > > > concern. > >>>>>> > > > > > >> > 1. In the past, the following buffers after the > >>>>>> barrier will > >>>>>> > > be > >>>>>> > > > > > >> cached on downstream side before alignment. > >>>>>> > > > > > >> > 2. In 1.11, the upstream would not send the buffers > >>>>>> after > >>>>>> > the > >>>>>> > > > > > >> barrier. When the downstream finishes the alignment, it > >>>>>> will > >>>>>> > > notify > >>>>>> > > > > the > >>>>>> > > > > > >> downstream of continuing sending following buffers, > >>>>>> since it can > >>>>>> > > > > process > >>>>>> > > > > > >> them after alignment. > >>>>>> > > > > > >> > 3. The only difference is that the temporary > blocked > >>>>>> buffers > >>>>>> > > are > >>>>>> > > > > > >> cached either on downstream side or on upstream side > >>>>>> before > >>>>>> > > > alignment. > >>>>>> > > > > > >> > 4. The side effect would be the additional > >>>>>> notification cost > >>>>>> > > for > >>>>>> > > > > > >> every barrier alignment. If the downstream and upstream > >>>>>> are > >>>>>> > > deployed > >>>>>> > > > > in > >>>>>> > > > > > >> separate TaskManager, the cost is network transport > >>>>>> delay (the > >>>>>> > > > effect > >>>>>> > > > > > can > >>>>>> > > > > > >> be ignored based on our testing with 1s checkpoint > >>>>>> interval). > >>>>>> > For > >>>>>> > > > > > sharing > >>>>>> > > > > > >> slot in your case, the cost is only one method call in > >>>>>> > processor, > >>>>>> > > > can > >>>>>> > > > > be > >>>>>> > > > > > >> ignored also. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > You mentioned "In this case, the downstream task > has > >>>>>> a high > >>>>>> > > > > average > >>>>>> > > > > > >> checkpoint duration(~30s, sync part)." This duration is > >>>>>> not > >>>>>> > > > reflecting > >>>>>> > > > > > the > >>>>>> > > > > > >> changes above, and it is only indicating the duration > for > >>>>>> > calling > >>>>>> > > > > > >> `Operation.snapshotState`. > >>>>>> > > > > > >> > If this duration is beyond your expectation, you > can > >>>>>> check > >>>>>> > or > >>>>>> > > > > debug > >>>>>> > > > > > >> whether the source/sink operations might take more time > >>>>>> to > >>>>>> > finish > >>>>>> > > > > > >> `snapshotState` in practice. E.g. you can > >>>>>> > > > > > >> > make the implementation of this method as empty to > >>>>>> further > >>>>>> > > > verify > >>>>>> > > > > > the > >>>>>> > > > > > >> effect. > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Best, > >>>>>> > > > > > >> > Zhijiang > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > >>>>>> ------------------------------------------------------------------ > >>>>>> > > > > > >> > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > Send Time:2020年7月5日(星期日) 12:22 > >>>>>> > > > > > >> > To:dev <dev@flink.apache.org>; Zhijiang < > >>>>>> > > > > wangzhijiang...@aliyun.com > >>>>>> > > > > > > > >>>>>> > > > > > >> > Cc:Yingjie Cao <kevin.ying...@gmail.com> > >>>>>> > > > > > >> > Subject:Re: [VOTE] Release 1.11.0, release > candidate > >>>>>> #4 > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Hi Zhijiang, > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Could you please point me to more details > regarding: > >>>>>> "[2]: > >>>>>> > > Delay > >>>>>> > > > > > send > >>>>>> > > > > > >> the > >>>>>> > > > > > >> > following buffers after checkpoint barrier on > >>>>>> upstream side > >>>>>> > > > until > >>>>>> > > > > > >> barrier > >>>>>> > > > > > >> > alignment on downstream side." > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > In this case, the downstream task has a high > average > >>>>>> > > checkpoint > >>>>>> > > > > > >> duration > >>>>>> > > > > > >> > (~30s, sync part). If there was a change to hold > >>>>>> buffers > >>>>>> > > > depending > >>>>>> > > > > > on > >>>>>> > > > > > >> > downstream performance, could this possibly apply > to > >>>>>> this > >>>>>> > case > >>>>>> > > > > (even > >>>>>> > > > > > >> when > >>>>>> > > > > > >> > there is no shuffle that would require alignment)? > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > Thanks, > >>>>>> > > > > > >> > Thomas > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > On Sat, Jul 4, 2020 at 7:39 AM Zhijiang < > >>>>>> > > > > wangzhijiang...@aliyun.com > >>>>>> > > > > > >> .invalid> > >>>>>> > > > > > >> > wrote: > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Hi Thomas, > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Thanks for the further update information. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > I guess we can dismiss the network stack changes, > >>>>>> since in > >>>>>> > > > your > >>>>>> > > > > > >> case the > >>>>>> > > > > > >> > > downstream and upstream would probably be > deployed > >>>>>> in the > >>>>>> > > same > >>>>>> > > > > > slot > >>>>>> > > > > > >> > > bypassing the network data shuffle. > >>>>>> > > > > > >> > > Also I guess release-1.11 will not bring general > >>>>>> > performance > >>>>>> > > > > > >> regression in > >>>>>> > > > > > >> > > runtime engine, as we also did the performance > >>>>>> testing for > >>>>>> > > all > >>>>>> > > > > > >> general > >>>>>> > > > > > >> > > cases by [1] in real cluster before and the > testing > >>>>>> > results > >>>>>> > > > > should > >>>>>> > > > > > >> fit the > >>>>>> > > > > > >> > > expectation. But we indeed did not test the > >>>>>> specific > >>>>>> > source > >>>>>> > > > and > >>>>>> > > > > > sink > >>>>>> > > > > > >> > > connectors yet as I known. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Regarding your performance regression with 40%, I > >>>>>> wonder > >>>>>> > it > >>>>>> > > is > >>>>>> > > > > > >> probably > >>>>>> > > > > > >> > > related to specific source/sink changes (e.g. > >>>>>> kinesis) or > >>>>>> > > > > > >> environment > >>>>>> > > > > > >> > > issues with corner case. > >>>>>> > > > > > >> > > If possible, it would be helpful to further > locate > >>>>>> whether > >>>>>> > > the > >>>>>> > > > > > >> regression > >>>>>> > > > > > >> > > is caused by kinesis, by replacing the kinesis > >>>>>> source & > >>>>>> > sink > >>>>>> > > > and > >>>>>> > > > > > >> keeping > >>>>>> > > > > > >> > > the others same. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > As you said, it would be efficient to contact > with > >>>>>> you > >>>>>> > > > directly > >>>>>> > > > > > >> next week > >>>>>> > > > > > >> > > to further discuss this issue. And we are > >>>>>> willing/eager to > >>>>>> > > > > provide > >>>>>> > > > > > >> any help > >>>>>> > > > > > >> > > to resolve this issue soon. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Besides that, I guess this issue should not be > the > >>>>>> blocker > >>>>>> > > for > >>>>>> > > > > the > >>>>>> > > > > > >> > > release, since it is probably a corner case based > >>>>>> on the > >>>>>> > > > current > >>>>>> > > > > > >> analysis. > >>>>>> > > > > > >> > > If we really conclude anything need to be > resolved > >>>>>> after > >>>>>> > the > >>>>>> > > > > final > >>>>>> > > > > > >> > > release, then we can also make the next minor > >>>>>> > release-1.11.1 > >>>>>> > > > > come > >>>>>> > > > > > >> soon. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > [1] > >>>>>> https://issues.apache.org/jira/browse/FLINK-18433 > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Best, > >>>>>> > > > > > >> > > Zhijiang > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >>>>>> ------------------------------------------------------------------ > >>>>>> > > > > > >> > > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > > Send Time:2020年7月4日(星期六) 12:26 > >>>>>> > > > > > >> > > To:dev <dev@flink.apache.org>; Zhijiang < > >>>>>> > > > > > wangzhijiang...@aliyun.com > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > Cc:Yingjie Cao <kevin.ying...@gmail.com> > >>>>>> > > > > > >> > > Subject:Re: [VOTE] Release 1.11.0, release > >>>>>> candidate #4 > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Hi Zhijiang, > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > It will probably be best if we connect next week > >>>>>> and > >>>>>> > discuss > >>>>>> > > > the > >>>>>> > > > > > >> issue > >>>>>> > > > > > >> > > directly since this could be quite difficult to > >>>>>> reproduce. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Before the testing result on our side comes out > >>>>>> for your > >>>>>> > > > > > respective > >>>>>> > > > > > >> job > >>>>>> > > > > > >> > > case, I have some other questions to confirm for > >>>>>> further > >>>>>> > > > > analysis: > >>>>>> > > > > > >> > > - How much percentage regression you found > >>>>>> after > >>>>>> > > > switching > >>>>>> > > > > to > >>>>>> > > > > > >> 1.11? > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > ~40% throughput decline > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > - Are there any network bottleneck in your > >>>>>> cluster? > >>>>>> > > E.g. > >>>>>> > > > > the > >>>>>> > > > > > >> network > >>>>>> > > > > > >> > > bandwidth is full caused by other jobs? If so, it > >>>>>> might > >>>>>> > have > >>>>>> > > > > more > >>>>>> > > > > > >> effects > >>>>>> > > > > > >> > > by above [2] > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > The test runs on a k8s cluster that is also used > >>>>>> for other > >>>>>> > > > > > >> production jobs. > >>>>>> > > > > > >> > > There is no reason be believe network is the > >>>>>> bottleneck. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > - Did you adjust the default network buffer > >>>>>> setting? > >>>>>> > > E.g. > >>>>>> > > > > > >> > > > >>>>>> "taskmanager.network.memory.floating-buffers-per-gate" or > >>>>>> > > > > > >> > > "taskmanager.network.memory.buffers-per-channel" > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > The job is using the defaults, i.e we don't > >>>>>> configure the > >>>>>> > > > > > settings. > >>>>>> > > > > > >> If you > >>>>>> > > > > > >> > > want me to try specific settings in the hope that > >>>>>> it will > >>>>>> > > help > >>>>>> > > > > to > >>>>>> > > > > > >> isolate > >>>>>> > > > > > >> > > the issue please let me know. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > - I guess the topology has three vertexes > >>>>>> > > > "KinesisConsumer > >>>>>> > > > > -> > >>>>>> > > > > > >> Chained > >>>>>> > > > > > >> > > FlatMap -> KinesisProducer", and the partition > >>>>>> mode for > >>>>>> > > > > > >> "KinesisConsumer -> > >>>>>> > > > > > >> > > FlatMap" and "FlatMap->KinesisProducer" are both > >>>>>> > "forward"? > >>>>>> > > If > >>>>>> > > > > so, > >>>>>> > > > > > >> the edge > >>>>>> > > > > > >> > > connection is one-to-one, not all-to-all, then > the > >>>>>> above > >>>>>> > > > [1][2] > >>>>>> > > > > > >> should no > >>>>>> > > > > > >> > > effects in theory with default network buffer > >>>>>> setting. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > There are only 2 vertices and the edge is > >>>>>> "forward". > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > - By slot sharing, I guess these three vertex > >>>>>> > > parallelism > >>>>>> > > > > task > >>>>>> > > > > > >> would > >>>>>> > > > > > >> > > probably be deployed into the same slot, then the > >>>>>> data > >>>>>> > > shuffle > >>>>>> > > > > is > >>>>>> > > > > > >> by memory > >>>>>> > > > > > >> > > queue, not network stack. If so, the above [2] > >>>>>> should no > >>>>>> > > > effect. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Yes, vertices share slots. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > - I also saw some Jira changes for kinesis in > >>>>>> this > >>>>>> > > > release, > >>>>>> > > > > > >> could you > >>>>>> > > > > > >> > > confirm that these changes would not effect the > >>>>>> > performance? > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > I will need to take a look. 1.10 already had a > >>>>>> regression > >>>>>> > > > > > >> introduced by the > >>>>>> > > > > > >> > > Kinesis producer update. > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > Thanks, > >>>>>> > > > > > >> > > Thomas > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang < > >>>>>> > > > > > >> wangzhijiang...@aliyun.com > >>>>>> > > > > > >> > > .invalid> > >>>>>> > > > > > >> > > wrote: > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > Hi Thomas, > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Thanks for your reply with rich information! > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > We are trying to reproduce your case in our > >>>>>> cluster to > >>>>>> > > > further > >>>>>> > > > > > >> verify it, > >>>>>> > > > > > >> > > > and @Yingjie Cao is working on it now. > >>>>>> > > > > > >> > > > As we have not kinesis consumer and producer > >>>>>> > internally, > >>>>>> > > so > >>>>>> > > > > we > >>>>>> > > > > > >> will > >>>>>> > > > > > >> > > > construct the common source and sink instead in > >>>>>> the case > >>>>>> > > of > >>>>>> > > > > > >> backpressure. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Firstly, we can dismiss the rockdb factor in > this > >>>>>> > release, > >>>>>> > > > > since > >>>>>> > > > > > >> you also > >>>>>> > > > > > >> > > > mentioned that "filesystem leads to same > >>>>>> symptoms". > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Secondly, if my understanding is right, you > >>>>>> emphasis > >>>>>> > that > >>>>>> > > > the > >>>>>> > > > > > >> regression > >>>>>> > > > > > >> > > > only exists for the jobs with low checkpoint > >>>>>> interval > >>>>>> > > (10s). > >>>>>> > > > > > >> > > > Based on that, I have two suspicions with the > >>>>>> network > >>>>>> > > > related > >>>>>> > > > > > >> changes in > >>>>>> > > > > > >> > > > this release: > >>>>>> > > > > > >> > > > - [1]: Limited the maximum backlog value > >>>>>> (default > >>>>>> > 10) > >>>>>> > > in > >>>>>> > > > > > >> subpartition > >>>>>> > > > > > >> > > > queue. > >>>>>> > > > > > >> > > > - [2]: Delay send the following buffers > after > >>>>>> > > checkpoint > >>>>>> > > > > > >> barrier on > >>>>>> > > > > > >> > > > upstream side until barrier alignment on > >>>>>> downstream > >>>>>> > side. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > These changes are motivated for reducing the > >>>>>> in-flight > >>>>>> > > > buffers > >>>>>> > > > > > to > >>>>>> > > > > > >> speedup > >>>>>> > > > > > >> > > > checkpoint especially in the case of > >>>>>> backpressure. > >>>>>> > > > > > >> > > > In theory they should have very minor > >>>>>> performance effect > >>>>>> > > and > >>>>>> > > > > > >> actually we > >>>>>> > > > > > >> > > > also tested in cluster to verify within > >>>>>> expectation > >>>>>> > before > >>>>>> > > > > > >> merging them, > >>>>>> > > > > > >> > > > but maybe there are other corner cases we have > >>>>>> not > >>>>>> > > thought > >>>>>> > > > of > >>>>>> > > > > > >> before. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Before the testing result on our side comes out > >>>>>> for your > >>>>>> > > > > > >> respective job > >>>>>> > > > > > >> > > > case, I have some other questions to confirm > for > >>>>>> further > >>>>>> > > > > > analysis: > >>>>>> > > > > > >> > > > - How much percentage regression you found > >>>>>> after > >>>>>> > > > > switching > >>>>>> > > > > > >> to 1.11? > >>>>>> > > > > > >> > > > - Are there any network bottleneck in your > >>>>>> cluster? > >>>>>> > > > E.g. > >>>>>> > > > > > the > >>>>>> > > > > > >> network > >>>>>> > > > > > >> > > > bandwidth is full caused by other jobs? If so, > >>>>>> it might > >>>>>> > > have > >>>>>> > > > > > more > >>>>>> > > > > > >> effects > >>>>>> > > > > > >> > > > by above [2] > >>>>>> > > > > > >> > > > - Did you adjust the default network > buffer > >>>>>> > setting? > >>>>>> > > > E.g. > >>>>>> > > > > > >> > > > > >>>>>> "taskmanager.network.memory.floating-buffers-per-gate" > >>>>>> > or > >>>>>> > > > > > >> > > > > "taskmanager.network.memory.buffers-per-channel" > >>>>>> > > > > > >> > > > - I guess the topology has three vertexes > >>>>>> > > > > "KinesisConsumer > >>>>>> > > > > > -> > >>>>>> > > > > > >> > > Chained > >>>>>> > > > > > >> > > > FlatMap -> KinesisProducer", and the partition > >>>>>> mode for > >>>>>> > > > > > >> "KinesisConsumer > >>>>>> > > > > > >> > > -> > >>>>>> > > > > > >> > > > FlatMap" and "FlatMap->KinesisProducer" are > both > >>>>>> > > "forward"? > >>>>>> > > > If > >>>>>> > > > > > >> so, the > >>>>>> > > > > > >> > > edge > >>>>>> > > > > > >> > > > connection is one-to-one, not all-to-all, then > >>>>>> the above > >>>>>> > > > > [1][2] > >>>>>> > > > > > >> should no > >>>>>> > > > > > >> > > > effects in theory with default network buffer > >>>>>> setting. > >>>>>> > > > > > >> > > > - By slot sharing, I guess these three > vertex > >>>>>> > > > parallelism > >>>>>> > > > > > >> task would > >>>>>> > > > > > >> > > > probably be deployed into the same slot, then > >>>>>> the data > >>>>>> > > > shuffle > >>>>>> > > > > > is > >>>>>> > > > > > >> by > >>>>>> > > > > > >> > > memory > >>>>>> > > > > > >> > > > queue, not network stack. If so, the above [2] > >>>>>> should no > >>>>>> > > > > effect. > >>>>>> > > > > > >> > > > - I also saw some Jira changes for kinesis > >>>>>> in this > >>>>>> > > > > release, > >>>>>> > > > > > >> could you > >>>>>> > > > > > >> > > > confirm that these changes would not effect the > >>>>>> > > performance? > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Best, > >>>>>> > > > > > >> > > > Zhijiang > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > > >>>>>> ------------------------------------------------------------------ > >>>>>> > > > > > >> > > > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > > > Send Time:2020年7月3日(星期五) 01:07 > >>>>>> > > > > > >> > > > To:dev <dev@flink.apache.org>; Zhijiang < > >>>>>> > > > > > >> wangzhijiang...@aliyun.com> > >>>>>> > > > > > >> > > > Subject:Re: [VOTE] Release 1.11.0, release > >>>>>> candidate #4 > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Hi Zhijiang, > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > The performance degradation manifests in > >>>>>> backpressure > >>>>>> > > which > >>>>>> > > > > > leads > >>>>>> > > > > > >> to > >>>>>> > > > > > >> > > > growing backlog in the source. I switched a few > >>>>>> times > >>>>>> > > > between > >>>>>> > > > > > >> 1.10 and > >>>>>> > > > > > >> > > 1.11 > >>>>>> > > > > > >> > > > and the behavior is consistent. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > The DAG is: > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > KinesisConsumer -> (Flat Map, Flat Map, Flat > Map) > >>>>>> > > -------- > >>>>>> > > > > > >> forward > >>>>>> > > > > > >> > > > ---------> KinesisProducer > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Parallelism: 160 > >>>>>> > > > > > >> > > > No shuffle/rebalance. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Checkpointing config: > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Checkpointing Mode Exactly Once > >>>>>> > > > > > >> > > > Interval 10s > >>>>>> > > > > > >> > > > Timeout 10m 0s > >>>>>> > > > > > >> > > > Minimum Pause Between Checkpoints 10s > >>>>>> > > > > > >> > > > Maximum Concurrent Checkpoints 1 > >>>>>> > > > > > >> > > > Persist Checkpoints Externally Enabled (delete > on > >>>>>> > > > > cancellation) > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > State backend: rocksdb (filesystem leads to > same > >>>>>> > > symptoms) > >>>>>> > > > > > >> > > > Checkpoint size is tiny (500KB) > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > An interesting difference to another job that I > >>>>>> had > >>>>>> > > upgraded > >>>>>> > > > > > >> successfully > >>>>>> > > > > > >> > > > is the low checkpointing interval. > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > Thanks, > >>>>>> > > > > > >> > > > Thomas > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang < > >>>>>> > > > > > >> wangzhijiang...@aliyun.com > >>>>>> > > > > > >> > > > .invalid> > >>>>>> > > > > > >> > > > wrote: > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > Hi Thomas, > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Thanks for the efficient feedback. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Regarding the suggestion of adding the > release > >>>>>> notes > >>>>>> > > > > document, > >>>>>> > > > > > >> I agree > >>>>>> > > > > > >> > > > > with your point. Maybe we should adjust the > >>>>>> vote > >>>>>> > > template > >>>>>> > > > > > >> accordingly > >>>>>> > > > > > >> > > in > >>>>>> > > > > > >> > > > > the respective wiki to guide the following > >>>>>> release > >>>>>> > > > > processes. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Regarding the performance regression, could > you > >>>>>> > provide > >>>>>> > > > some > >>>>>> > > > > > >> more > >>>>>> > > > > > >> > > details > >>>>>> > > > > > >> > > > > for our better measurement or reproducing on > >>>>>> our > >>>>>> > sides? > >>>>>> > > > > > >> > > > > E.g. I guess the topology only includes two > >>>>>> vertexes > >>>>>> > > > source > >>>>>> > > > > > and > >>>>>> > > > > > >> sink? > >>>>>> > > > > > >> > > > > What is the parallelism for every vertex? > >>>>>> > > > > > >> > > > > The upstream shuffles data to the downstream > >>>>>> via > >>>>>> > > rebalance > >>>>>> > > > > > >> partitioner > >>>>>> > > > > > >> > > or > >>>>>> > > > > > >> > > > > other? > >>>>>> > > > > > >> > > > > The checkpoint mode is exactly-once with > >>>>>> rocksDB state > >>>>>> > > > > > backend? > >>>>>> > > > > > >> > > > > The backpressure happened in this case? > >>>>>> > > > > > >> > > > > How much percentage regression in this case? > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Best, > >>>>>> > > > > > >> > > > > Zhijiang > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > >>>>>> > ------------------------------------------------------------------ > >>>>>> > > > > > >> > > > > From:Thomas Weise <t...@apache.org> > >>>>>> > > > > > >> > > > > Send Time:2020年7月2日(星期四) 09:54 > >>>>>> > > > > > >> > > > > To:dev <dev@flink.apache.org> > >>>>>> > > > > > >> > > > > Subject:Re: [VOTE] Release 1.11.0, release > >>>>>> candidate > >>>>>> > #4 > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Hi Till, > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Yes, we don't have the setting in > >>>>>> flink-conf.yaml. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Generally, we carry forward the existing > >>>>>> configuration > >>>>>> > > and > >>>>>> > > > > any > >>>>>> > > > > > >> change > >>>>>> > > > > > >> > > to > >>>>>> > > > > > >> > > > > default configuration values would impact the > >>>>>> upgrade. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Yes, since it is an incompatible change I > >>>>>> would state > >>>>>> > it > >>>>>> > > > in > >>>>>> > > > > > the > >>>>>> > > > > > >> release > >>>>>> > > > > > >> > > > > notes. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > Thanks, > >>>>>> > > > > > >> > > > > Thomas > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > BTW I found a performance regression while > >>>>>> trying to > >>>>>> > > > upgrade > >>>>>> > > > > > >> another > >>>>>> > > > > > >> > > > > pipeline with this RC. It is a simple Kinesis > >>>>>> to > >>>>>> > Kinesis > >>>>>> > > > > job. > >>>>>> > > > > > >> Wasn't > >>>>>> > > > > > >> > > able > >>>>>> > > > > > >> > > > > to pin it down yet, symptoms include > increased > >>>>>> > > checkpoint > >>>>>> > > > > > >> alignment > >>>>>> > > > > > >> > > time. > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > On Wed, Jul 1, 2020 at 12:04 AM Till > Rohrmann < > >>>>>> > > > > > >> trohrm...@apache.org> > >>>>>> > > > > > >> > > > > wrote: > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > Hi Thomas, > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > just to confirm: When starting the image in > >>>>>> local > >>>>>> > > mode, > >>>>>> > > > > then > >>>>>> > > > > > >> you > >>>>>> > > > > > >> > > don't > >>>>>> > > > > > >> > > > > have > >>>>>> > > > > > >> > > > > > any of the JobManager memory configuration > >>>>>> settings > >>>>>> > > > > > >> configured in the > >>>>>> > > > > > >> > > > > > effective flink-conf.yaml, right? Does this > >>>>>> mean > >>>>>> > that > >>>>>> > > > you > >>>>>> > > > > > have > >>>>>> > > > > > >> > > > explicitly > >>>>>> > > > > > >> > > > > > removed `jobmanager.heap.size: 1024m` from > >>>>>> the > >>>>>> > default > >>>>>> > > > > > >> configuration? > >>>>>> > > > > > >> > > > If > >>>>>> > > > > > >> > > > > > this is the case, then I believe it was > more > >>>>>> of an > >>>>>> > > > > > >> unintentional > >>>>>> > > > > > >> > > > artifact > >>>>>> > > > > > >> > > > > > that it worked before and it has been > >>>>>> corrected now > >>>>>> > so > >>>>>> > > > > that > >>>>>> > > > > > >> one needs > >>>>>> > > > > > >> > > > to > >>>>>> > > > > > >> > > > > > specify the memory of the JM process > >>>>>> explicitly. Do > >>>>>> > > you > >>>>>> > > > > > think > >>>>>> > > > > > >> it > >>>>>> > > > > > >> > > would > >>>>>> > > > > > >> > > > > help > >>>>>> > > > > > >> > > > > > to explicitly state this in the release > >>>>>> notes? > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > Cheers, > >>>>>> > > > > > >> > > > > > Till > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas > Weise < > >>>>>> > > > > t...@apache.org > >>>>>> > > > > > > > >>>>>> > > > > > >> wrote: > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > > Thanks for preparing another RC! > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > As mentioned in the previous RC thread, > it > >>>>>> would > >>>>>> > be > >>>>>> > > > > super > >>>>>> > > > > > >> helpful > >>>>>> > > > > > >> > > if > >>>>>> > > > > > >> > > > > the > >>>>>> > > > > > >> > > > > > > release notes that are part of the > >>>>>> documentation > >>>>>> > can > >>>>>> > > > be > >>>>>> > > > > > >> included > >>>>>> > > > > > >> > > [1]. > >>>>>> > > > > > >> > > > > > It's > >>>>>> > > > > > >> > > > > > > a significant time-saver to have read > >>>>>> those first. > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > I found one more non-backward compatible > >>>>>> change > >>>>>> > that > >>>>>> > > > > would > >>>>>> > > > > > >> be worth > >>>>>> > > > > > >> > > > > > > addressing/mentioning: > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > It is now necessary to configure the > >>>>>> jobmanager > >>>>>> > heap > >>>>>> > > > > size > >>>>>> > > > > > in > >>>>>> > > > > > >> > > > > > > flink-conf.yaml (with either > >>>>>> jobmanager.heap.size > >>>>>> > > > > > >> > > > > > > or jobmanager.memory.heap.size). Why > would > >>>>>> I not > >>>>>> > > want > >>>>>> > > > to > >>>>>> > > > > > do > >>>>>> > > > > > >> that > >>>>>> > > > > > >> > > > > anyways? > >>>>>> > > > > > >> > > > > > > Well, we set it dynamically for a cluster > >>>>>> > deployment > >>>>>> > > > via > >>>>>> > > > > > the > >>>>>> > > > > > >> > > > > > > flinkk8soperator, but the container image > >>>>>> can also > >>>>>> > > be > >>>>>> > > > > used > >>>>>> > > > > > >> for > >>>>>> > > > > > >> > > > testing > >>>>>> > > > > > >> > > > > > with > >>>>>> > > > > > >> > > > > > > local mode (./bin/jobmanager.sh > >>>>>> start-foreground > >>>>>> > > > local). > >>>>>> > > > > > >> That will > >>>>>> > > > > > >> > > > fail > >>>>>> > > > > > >> > > > > > if > >>>>>> > > > > > >> > > > > > > the heap wasn't configured and that's > how I > >>>>>> > noticed > >>>>>> > > > it. > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > Thanks, > >>>>>> > > > > > >> > > > > > > Thomas > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > [1] > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang > < > >>>>>> > > > > > >> > > wangzhijiang...@aliyun.com > >>>>>> > > > > > >> > > > > > > .invalid> > >>>>>> > > > > > >> > > > > > > wrote: > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > > Hi everyone, > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > Please review and vote on the release > >>>>>> candidate > >>>>>> > #4 > >>>>>> > > > for > >>>>>> > > > > > the > >>>>>> > > > > > >> > > version > >>>>>> > > > > > >> > > > > > > 1.11.0, > >>>>>> > > > > > >> > > > > > > > as follows: > >>>>>> > > > > > >> > > > > > > > [ ] +1, Approve the release > >>>>>> > > > > > >> > > > > > > > [ ] -1, Do not approve the release > >>>>>> (please > >>>>>> > provide > >>>>>> > > > > > >> specific > >>>>>> > > > > > >> > > > comments) > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > The complete staging area is available > >>>>>> for your > >>>>>> > > > > review, > >>>>>> > > > > > >> which > >>>>>> > > > > > >> > > > > includes: > >>>>>> > > > > > >> > > > > > > > * JIRA release notes [1], > >>>>>> > > > > > >> > > > > > > > * the official Apache source release > and > >>>>>> binary > >>>>>> > > > > > >> convenience > >>>>>> > > > > > >> > > > releases > >>>>>> > > > > > >> > > > > to > >>>>>> > > > > > >> > > > > > > be > >>>>>> > > > > > >> > > > > > > > deployed to dist.apache.org [2], which > >>>>>> are > >>>>>> > signed > >>>>>> > > > > with > >>>>>> > > > > > >> the key > >>>>>> > > > > > >> > > > with > >>>>>> > > > > > >> > > > > > > > fingerprint > >>>>>> > > 2DA85B93244FDFA19A6244500653C0A2CEA00D0E > >>>>>> > > > > > [3], > >>>>>> > > > > > >> > > > > > > > * all artifacts to be deployed to the > >>>>>> Maven > >>>>>> > > Central > >>>>>> > > > > > >> Repository > >>>>>> > > > > > >> > > [4], > >>>>>> > > > > > >> > > > > > > > * source code tag "release-1.11.0-rc4" > >>>>>> [5], > >>>>>> > > > > > >> > > > > > > > * website pull request listing the new > >>>>>> release > >>>>>> > and > >>>>>> > > > > > adding > >>>>>> > > > > > >> > > > > announcement > >>>>>> > > > > > >> > > > > > > > blog post [6]. > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > The vote will be open for at least 72 > >>>>>> hours. It > >>>>>> > is > >>>>>> > > > > > >> adopted by > >>>>>> > > > > > >> > > > > majority > >>>>>> > > > > > >> > > > > > > > approval, with at least 3 PMC > >>>>>> affirmative votes. > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > Thanks, > >>>>>> > > > > > >> > > > > > > > Release Manager > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > [1] > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364 > >>>>>> > > > > > >> > > > > > > > [2] > >>>>>> > > > > > >> > > > >>>>>> > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/ > >>>>>> > > > > > >> > > > > > > > [3] > >>>>>> > > > > > https://dist.apache.org/repos/dist/release/flink/KEYS > >>>>>> > > > > > >> > > > > > > > [4] > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > >>>>>> > > > > > >>>>>> > > > >>>>>> > https://repository.apache.org/content/repositories/orgapacheflink-1377/ > >>>>>> > > > > > >> > > > > > > > [5] > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >>>>>> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4 > >>>>>> > > > > > >> > > > > > > > [6] > >>>>>> > https://github.com/apache/flink-web/pull/352 > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > > >>>>>> > > > > > >> > > > > > > > >>>>>> > > > > > >> > > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > > >>>>>> > > > > > >> > >>>>>> > > > > > >> > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > >>>>>> > > >>>>>> > -- > >>>>>> > Regards, > >>>>>> > Roman > >>>>>> > > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> Roman > >>>>> > >>>> > >> > >> -- > >> Regards, > >> Roman > >> > > > > > > -- > > Regards, > > Roman > > >
