Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[柳尘]

Thanks, Yanfei, to drive this and make the performance monitoring publicly
available.

I'd like to join as a maintainer.
Looking forward to the workflow.

Best
xingyuan




yanfei lei  于2022年10月26日周三 11:33写道：

> Hi everyone,
>
> As discussed earlier, we plan to create a benchmark channel in Apache Flink
> slack[1], but the plan was shelved for a while[2]. So I went on with this
> work, and created the #flink-dev-benchmarks channel for performance
> regression notifications.
>
> We have a regression report script[3] that runs daily, and a notification
> would be sent to the slack channel when the last few benchmark results are
> significantly worse than the baseline.
> Note, regressions are detected by a simple script which may have false
> positives and false negatives. And all benchmarks are executed on one
> physical machine[4] which is provided by Ververica(Alibaba)[5], it might
> happen that hardware issues affect performance, like "[FLINK-18614
> ] Performance
> regression
> 2020.07.13"[6].
>
> After the migration, we need a procedure to watch over the entire
> performance of Flink code together. For example, if a regression
> occurs, investigating the cause and resolving the problem are needed. In
> the past, this procedure is maintained internally within Ververica, but we
> think making the procedure public would benefit all. I volunteer to serve
> as one of the initial maintainers, and would be glad if more contributors
> can join me. I'd also prepare some guidelines to help others get familiar
> with the workflow. I will start a new thread to discuss the workflow soon.
>
>
> [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> [2] https://issues.apache.org/jira/browse/FLINK-28468
> [3]
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> [4] http://codespeed.dak8s.net:8080
> [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
>
> [6] https://issues.apache.org/jira/browse/FLINK-18614
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Matthias Pohl]

ng time, see [5]
> > > > >> > >https://issues.apache.org/jira/browse/FLINK-27165 for
> > possible
> > > > >> > reasons.
> > > > >> > >2. Regressions are detected by a simple script which may
> have
> > > > false
> > > > >> > >positives and false negatives, especially for benchmarks
> with
> > > > small
> > > > >> > >absolute values, small value changes cause large percentage
> > > > changes.
> > > > >> > see
> > > > >> > >[6] for details.
> > > > >> > >
> > > > >> > >  Maybe slidingWindow
> > > > >> > > <
> > > > >> > >
> > > > >> >
> > > >
> > >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=slidingWindow=on=on=off=2=200
> > > > >> > > >(value~=600),
> > > > >> > > stateBackends.ROCKS
> > > > >> > > <
> > > > >> > >
> > > > >> >
> > > >
> > >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=stateBackends.ROCKS=on=on=off=2=200
> > > > >> > > >
> > > > >> > > (value~=260) and serializerHeavyString
> > > > >> > > <
> > > > >> > >
> > > > >> >
> > > >
> > >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=serializerHeavyString=on=on=off=2=200
> > > > >> > > >(value~=170)
> > > > >> > > are
> > > > >> > > not true regressions.
> > > > >> > >
> > > > >> > >1. For deployAllTasks.STREAMING
> > > > >> > ><
> > > > >> > >
> > > > >> >
> > > >
> > >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=8=deployAllTasks.STREAMING=on=on=off=2=200
> > > > >> > > >,
> > > > >> > >this benchmark result is how much time it takes to deploy
> > job,
> > > > the
> > > > >> > less
> > > > >> > >value the better performance, see [7] for details.
> > FLINK-27571
> > > > >> > ><https://issues.apache.org/jira/browse/FLINK-27571>[8]
> would
> > > > fix this
> > > > >> > >problem.
> > > > >> > >
> > > > >> > >
> > > > >> > > As mentioned before, regressions are detected by a simple
> script
> > > > that is
> > > > >> > > less stable, FLINK-29825 <
> > > > >> > > https://issues.apache.org/jira/browse/FLINK-29825>[9]
> > > > >> > > is created to improve the benchmark's stability. I planned to
> > > > invite more
> > > > >> > > volunteers to monitor it after the checking of regression
> became
> > > > more
> > > > >> > > stable, but I've been stuck with something else lately, sorry
> > for
> > > > the
> > > > >> > late
> > > > >> > > response.  Any suggestions on handling benchmark
> > regressions/fails
> > > > are
> > > > >> > > welcome.
> > > > >> > >
> > > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-29883
> > > > >> > >
> > > > >> > > [2] https://issues.apache.org/jira/browse/FLINK-29886
> > > > >> > >
> > > > >> > > [3] https://issues.apache.org/jira/browse/FLINK-30015
> > > > >> > >
> > > > >> > > [4] https://issues.apache.org/jira/browse/FLINK-30181
> > > > >> > >
> > > > >> > > [5] https://issues.apache.org/jira/browse/FLINK-27165
> > > > >> > >
> > > > >> > > [6]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> > > > >> > >
> > > > >> > > [7]
> > > > >> > >
> > > > >>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Martijn Visser]

nchmark's stability. I planned to
> > > invite more
> > > >> > > volunteers to monitor it after the checking of regression became
> > > more
> > > >> > > stable, but I've been stuck with something else lately, sorry
> for
> > > the
> > > >> > late
> > > >> > > response.  Any suggestions on handling benchmark
> regressions/fails
> > > are
> > > >> > > welcome.
> > > >> > >
> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-29883
> > > >> > >
> > > >> > > [2] https://issues.apache.org/jira/browse/FLINK-29886
> > > >> > >
> > > >> > > [3] https://issues.apache.org/jira/browse/FLINK-30015
> > > >> > >
> > > >> > > [4] https://issues.apache.org/jira/browse/FLINK-30181
> > > >> > >
> > > >> > > [5] https://issues.apache.org/jira/browse/FLINK-27165
> > > >> > >
> > > >> > > [6]
> > > >> > >
> > > >> > >
> > > >> >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> > > >> > >
> > > >> > > [7]
> > > >> > >
> > > >> > >
> > > >> >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58
> > > >> > >
> > > >> > > [8] https://issues.apache.org/jira/browse/FLINK-27571
> > > >> > >
> > > >> > > [9] https://issues.apache.org/jira/browse/FLINK-29825
> > > >> > >
> > > >> > >
> > > >> > > Best,
> > > >> > >
> > > >> > > Yanfei
> > > >> > >
> > > >> > > Martijn Visser  于2022年11月29日周二
> 15:54写道：
> > > >> > >
> > > >> > > > Hi,
> > > >> > > >
> > > >> > > > Is there any update to be expected on the benchmark? I see
> > > results of
> > > >> > the
> > > >> > > > benchmark being posted to Slack, but it appears that it's not
> > > being
> > > >> > > > monitored and no follow-up actions are being taken. I think
> it's
> > > >> > > currently
> > > >> > > > lacking a process on how to interpret the results and what
> > action
> > > >> > should
> > > >> > > > be taken and by whom.
> > > >> > > >
> > > >> > > > Best regards,
> > > >> > > >
> > > >> > > > Martijn
> > > >> > > >
> > > >> > > > On Thu, Nov 3, 2022 at 12:22 PM Jing Ge 
> > > wrote:
> > > >> > > >
> > > >> > > > > Thanks yanfei for driving this!
> > > >> > > > >
> > > >> > > > > Looking forward to further discussion w.r.t. the workflow.
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > > Jing
> > > >> > > > >
> > > >> > > > > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen <
> > > mas.chen6...@gmail.com>
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > > +1, thanks for driving this!
> > > >> > > > > >
> > > >> > > > > > On a side note, can we also ensure that a performance
> > summary
> > > >> > report
> > > >> > > > for
> > > >> > > > > > Flink major version upgrades is in release notes, once
> this
> > > >> > > > > infrastructure
> > > >> > > > > > becomes mature? From the user perspective, it would be
> nice
> > > to know
> > > >> > > > what
> > > >> > > > > > the expected (or unexpected) regressions in a major
> version
> > > upgrade
> > > >> > > > are.
> > > >> > > > > > I've seen the community do something like this before
> (

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yuan Mei]

t; > >> > >It is unstable for a long time, see [5]
> > >> > >https://issues.apache.org/jira/browse/FLINK-27165 for possible
> > >> > reasons.
> > >> > >2. Regressions are detected by a simple script which may have
> > false
> > >> > >positives and false negatives, especially for benchmarks with
> > small
> > >> > >absolute values, small value changes cause large percentage
> > changes.
> > >> > see
> > >> > >[6] for details.
> > >> > >
> > >> > >  Maybe slidingWindow
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=slidingWindow=on=on=off=2=200
> > >> > > >(value~=600),
> > >> > > stateBackends.ROCKS
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=stateBackends.ROCKS=on=on=off=2=200
> > >> > > >
> > >> > > (value~=260) and serializerHeavyString
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=serializerHeavyString=on=on=off=2=200
> > >> > > >(value~=170)
> > >> > > are
> > >> > > not true regressions.
> > >> > >
> > >> > >1. For deployAllTasks.STREAMING
> > >> > ><
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=8=deployAllTasks.STREAMING=on=on=off=2=200
> > >> > > >,
> > >> > >this benchmark result is how much time it takes to deploy job,
> > the
> > >> > less
> > >> > >value the better performance, see [7] for details. FLINK-27571
> > >> > ><https://issues.apache.org/jira/browse/FLINK-27571>[8] would
> > fix this
> > >> > >problem.
> > >> > >
> > >> > >
> > >> > > As mentioned before, regressions are detected by a simple script
> > that is
> > >> > > less stable, FLINK-29825 <
> > >> > > https://issues.apache.org/jira/browse/FLINK-29825>[9]
> > >> > > is created to improve the benchmark's stability. I planned to
> > invite more
> > >> > > volunteers to monitor it after the checking of regression became
> > more
> > >> > > stable, but I've been stuck with something else lately, sorry for
> > the
> > >> > late
> > >> > > response.  Any suggestions on handling benchmark regressions/fails
> > are
> > >> > > welcome.
> > >> > >
> > >> > > [1] https://issues.apache.org/jira/browse/FLINK-29883
> > >> > >
> > >> > > [2] https://issues.apache.org/jira/browse/FLINK-29886
> > >> > >
> > >> > > [3] https://issues.apache.org/jira/browse/FLINK-30015
> > >> > >
> > >> > > [4] https://issues.apache.org/jira/browse/FLINK-30181
> > >> > >
> > >> > > [5] https://issues.apache.org/jira/browse/FLINK-27165
> > >> > >
> > >> > > [6]
> > >> > >
> > >> > >
> > >> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> > >> > >
> > >> > > [7]
> > >> > >
> > >> > >
> > >> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58
> > >> > >
> > >> > > [8] https://issues.apache.org/jira/browse/FLINK-27571
> > >> > >
> > >> > > [9] https://issues.apache.org/jira/browse/FLINK-29825
> > >> > >
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Yanfei
> > >> > >
> > >> > > Martijn Visser  于2022年11月29日周二 15:54写道：
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > Is there any update to be expected on the benchmark? I see
> > results of
> > >> > the
> > >

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Qingsheng Ren]

he checking of regression became
> more
> >> > > stable, but I've been stuck with something else lately, sorry for
> the
> >> > late
> >> > > response.  Any suggestions on handling benchmark regressions/fails
> are
> >> > > welcome.
> >> > >
> >> > > [1] https://issues.apache.org/jira/browse/FLINK-29883
> >> > >
> >> > > [2] https://issues.apache.org/jira/browse/FLINK-29886
> >> > >
> >> > > [3] https://issues.apache.org/jira/browse/FLINK-30015
> >> > >
> >> > > [4] https://issues.apache.org/jira/browse/FLINK-30181
> >> > >
> >> > > [5] https://issues.apache.org/jira/browse/FLINK-27165
> >> > >
> >> > > [6]
> >> > >
> >> > >
> >> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> >> > >
> >> > > [7]
> >> > >
> >> > >
> >> >
> https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58
> >> > >
> >> > > [8] https://issues.apache.org/jira/browse/FLINK-27571
> >> > >
> >> > > [9] https://issues.apache.org/jira/browse/FLINK-29825
> >> > >
> >> > >
> >> > > Best,
> >> > >
> >> > > Yanfei
> >> > >
> >> > > Martijn Visser  于2022年11月29日周二 15:54写道：
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > Is there any update to be expected on the benchmark? I see
> results of
> >> > the
> >> > > > benchmark being posted to Slack, but it appears that it's not
> being
> >> > > > monitored and no follow-up actions are being taken. I think it's
> >> > > currently
> >> > > > lacking a process on how to interpret the results and what action
> >> > should
> >> > > > be taken and by whom.
> >> > > >
> >> > > > Best regards,
> >> > > >
> >> > > > Martijn
> >> > > >
> >> > > > On Thu, Nov 3, 2022 at 12:22 PM Jing Ge 
> wrote:
> >> > > >
> >> > > > > Thanks yanfei for driving this!
> >> > > > >
> >> > > > > Looking forward to further discussion w.r.t. the workflow.
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Jing
> >> > > > >
> >> > > > > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen <
> mas.chen6...@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > +1, thanks for driving this!
> >> > > > > >
> >> > > > > > On a side note, can we also ensure that a performance summary
> >> > report
> >> > > > for
> >> > > > > > Flink major version upgrades is in release notes, once this
> >> > > > > infrastructure
> >> > > > > > becomes mature? From the user perspective, it would be nice
> to know
> >> > > > what
> >> > > > > > the expected (or unexpected) regressions in a major version
> upgrade
> >> > > > are.
> >> > > > > > I've seen the community do something like this before (e.g.
> the
> >> > major
> >> > > > > > rocksdb version bump in 1.14?) and it was quite valuable to
> know
> >> > that
> >> > > > > > upfront!
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Mason
> >> > > > > >
> >> > > > > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo <
> >> > > guoweijieres...@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks Yanfei for driving this.
> >> > > > > > >
> >> > > > > > > It allows us to easily find the problem of performance
> >> > regression.
> >> > > > > > > Especially recently, I have made some improvements to the
> >> > > sched

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yanfei Lei]

; > Martijn
>> > > >
>> > > > On Thu, Nov 3, 2022 at 12:22 PM Jing Ge  wrote:
>> > > >
>> > > > > Thanks yanfei for driving this!
>> > > > >
>> > > > > Looking forward to further discussion w.r.t. the workflow.
>> > > > >
>> > > > > Best regards,
>> > > > > Jing
>> > > > >
>> > > > > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen 
>> > > > wrote:
>> > > > >
>> > > > > > +1, thanks for driving this!
>> > > > > >
>> > > > > > On a side note, can we also ensure that a performance summary
>> > report
>> > > > for
>> > > > > > Flink major version upgrades is in release notes, once this
>> > > > > infrastructure
>> > > > > > becomes mature? From the user perspective, it would be nice to know
>> > > > what
>> > > > > > the expected (or unexpected) regressions in a major version upgrade
>> > > > are.
>> > > > > > I've seen the community do something like this before (e.g. the
>> > major
>> > > > > > rocksdb version bump in 1.14?) and it was quite valuable to know
>> > that
>> > > > > > upfront!
>> > > > > >
>> > > > > > Best,
>> > > > > > Mason
>> > > > > >
>> > > > > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo <
>> > > guoweijieres...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks Yanfei for driving this.
>> > > > > > >
>> > > > > > > It allows us to easily find the problem of performance
>> > regression.
>> > > > > > > Especially recently, I have made some improvements to the
>> > > scheduling
>> > > > > > > related parts, your work is very important to ensure that these
>> > > > changes
>> > > > > > do
>> > > > > > > not cause some unexpected problems.
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > >
>> > > > > > > Weijie
>> > > > > > >
>> > > > > > >
>> > > > > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
>> > > > > > >
>> > > > > > > > Thanks for driving this and making the performance monitoring
>> > > > public,
>> > > > > > > this
>> > > > > > > > can make us know and resolve the performance problem quickly.
>> > > > > > > >
>> > > > > > > > Looking forward to the workflow and detailed descriptions fo
>> > > > > > > > flink-dev-benchmarks.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Congxian
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Yun Tang  于2022年10月27日周四 12:41写道：
>> > > > > > > >
>> > > > > > > > > Thanks, Yanfei for driving this to monitor the performance in
>> > > the
>> > > > > > > Apache
>> > > > > > > > > Flink Slack Channel.
>> > > > > > > > >
>> > > > > > > > > Look forward to the workflow and detailed descriptions of
>> > > > > > > > > flink-dev-benchmarks.
>> > > > > > > > >
>> > > > > > > > > Best
>> > > > > > > > > Yun Tang
>> > > > > > > > > 
>> > > > > > > > > From: Hangxiang Yu 
>> > > > > > > > > Sent: Thursday, October 27, 2022 10:59
>> > > > > > > > > To: dev@flink.apache.org 
>> > > > > > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved
>> > from
>> > > > > > > Ververica
>> > > > > > > > > to Apache Flink Slack Channel
>> > > > > > > > >
>> > > > > > > > > Hi, 

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yanfei Lei]

n Mon, Oct 31, 2022 at 6:04 PM Mason Chen  >
> > > > wrote:
> > > > >
> > > > > > +1, thanks for driving this!
> > > > > >
> > > > > > On a side note, can we also ensure that a performance summary
> > report
> > > > for
> > > > > > Flink major version upgrades is in release notes, once this
> > > > > infrastructure
> > > > > > becomes mature? From the user perspective, it would be nice to
> know
> > > > what
> > > > > > the expected (or unexpected) regressions in a major version
> upgrade
> > > > are.
> > > > > > I've seen the community do something like this before (e.g. the
> > major
> > > > > > rocksdb version bump in 1.14?) and it was quite valuable to know
> > that
> > > > > > upfront!
> > > > > >
> > > > > > Best,
> > > > > > Mason
> > > > > >
> > > > > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo <
> > > guoweijieres...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Yanfei for driving this.
> > > > > > >
> > > > > > > It allows us to easily find the problem of performance
> > regression.
> > > > > > > Especially recently, I have made some improvements to the
> > > scheduling
> > > > > > > related parts, your work is very important to ensure that these
> > > > changes
> > > > > > do
> > > > > > > not cause some unexpected problems.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Weijie
> > > > > > >
> > > > > > >
> > > > > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > > > > > >
> > > > > > > > Thanks for driving this and making the performance monitoring
> > > > public,
> > > > > > > this
> > > > > > > > can make us know and resolve the performance problem quickly.
> > > > > > > >
> > > > > > > > Looking forward to the workflow and detailed descriptions fo
> > > > > > > > flink-dev-benchmarks.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Congxian
> > > > > > > >
> > > > > > > >
> > > > > > > > Yun Tang  于2022年10月27日周四 12:41写道：
> > > > > > > >
> > > > > > > > > Thanks, Yanfei for driving this to monitor the performance
> in
> > > the
> > > > > > > Apache
> > > > > > > > > Flink Slack Channel.
> > > > > > > > >
> > > > > > > > > Look forward to the workflow and detailed descriptions of
> > > > > > > > > flink-dev-benchmarks.
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Yun Tang
> > > > > > > > > 
> > > > > > > > > From: Hangxiang Yu 
> > > > > > > > > Sent: Thursday, October 27, 2022 10:59
> > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved
> > from
> > > > > > > Ververica
> > > > > > > > > to Apache Flink Slack Channel
> > > > > > > > >
> > > > > > > > > Hi, Yanfei.
> > > > > > > > > Thanks for driving this.
> > > > > > > > > It could help us to detect and resolve the regression
> problem
> > > > > quickly
> > > > > > > and
> > > > > > > > > officially.
> > > > > > > > > I'd like to join as a maintainer.
> > > > > > > > > Looking forward to the workflow.
> > > > > > > > >
> > > > > > > > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei <
> > > yuanmei.w...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > 

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Martijn Visser]

hing else lately, sorry for the
> late
> > response.  Any suggestions on handling benchmark regressions/fails are
> > welcome.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-29883
> >
> > [2] https://issues.apache.org/jira/browse/FLINK-29886
> >
> > [3] https://issues.apache.org/jira/browse/FLINK-30015
> >
> > [4] https://issues.apache.org/jira/browse/FLINK-30181
> >
> > [5] https://issues.apache.org/jira/browse/FLINK-27165
> >
> > [6]
> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> >
> > [7]
> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58
> >
> > [8] https://issues.apache.org/jira/browse/FLINK-27571
> >
> > [9] https://issues.apache.org/jira/browse/FLINK-29825
> >
> >
> > Best,
> >
> > Yanfei
> >
> > Martijn Visser  于2022年11月29日周二 15:54写道：
> >
> > > Hi,
> > >
> > > Is there any update to be expected on the benchmark? I see results of
> the
> > > benchmark being posted to Slack, but it appears that it's not being
> > > monitored and no follow-up actions are being taken. I think it's
> > currently
> > > lacking a process on how to interpret the results and what action
> should
> > > be taken and by whom.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Thu, Nov 3, 2022 at 12:22 PM Jing Ge  wrote:
> > >
> > > > Thanks yanfei for driving this!
> > > >
> > > > Looking forward to further discussion w.r.t. the workflow.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen 
> > > wrote:
> > > >
> > > > > +1, thanks for driving this!
> > > > >
> > > > > On a side note, can we also ensure that a performance summary
> report
> > > for
> > > > > Flink major version upgrades is in release notes, once this
> > > > infrastructure
> > > > > becomes mature? From the user perspective, it would be nice to know
> > > what
> > > > > the expected (or unexpected) regressions in a major version upgrade
> > > are.
> > > > > I've seen the community do something like this before (e.g. the
> major
> > > > > rocksdb version bump in 1.14?) and it was quite valuable to know
> that
> > > > > upfront!
> > > > >
> > > > > Best,
> > > > > Mason
> > > > >
> > > > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo <
> > guoweijieres...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Yanfei for driving this.
> > > > > >
> > > > > > It allows us to easily find the problem of performance
> regression.
> > > > > > Especially recently, I have made some improvements to the
> > scheduling
> > > > > > related parts, your work is very important to ensure that these
> > > changes
> > > > > do
> > > > > > not cause some unexpected problems.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Weijie
> > > > > >
> > > > > >
> > > > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > > > > >
> > > > > > > Thanks for driving this and making the performance monitoring
> > > public,
> > > > > > this
> > > > > > > can make us know and resolve the performance problem quickly.
> > > > > > >
> > > > > > > Looking forward to the workflow and detailed descriptions fo
> > > > > > > flink-dev-benchmarks.
> > > > > > >
> > > > > > > Best,
> > > > > > > Congxian
> > > > > > >
> > > > > > >
> > > > > > > Yun Tang  于2022年10月27日周四 12:41写道：
> > > > > > >
> > > > > > > > Thanks, Yanfei for driving this to monitor the performance in
> > the
> > > > > > Apache
> > > > > > > > Flink Slack Channel.
> > > > > > > >
> > &

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Matthias Pohl]

d and no follow-up actions are being taken. I think it's
> currently
> > lacking a process on how to interpret the results and what action should
> > be taken and by whom.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Thu, Nov 3, 2022 at 12:22 PM Jing Ge  wrote:
> >
> > > Thanks yanfei for driving this!
> > >
> > > Looking forward to further discussion w.r.t. the workflow.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Mon, Oct 31, 2022 at 6:04 PM Mason Chen 
> > wrote:
> > >
> > > > +1, thanks for driving this!
> > > >
> > > > On a side note, can we also ensure that a performance summary report
> > for
> > > > Flink major version upgrades is in release notes, once this
> > > infrastructure
> > > > becomes mature? From the user perspective, it would be nice to know
> > what
> > > > the expected (or unexpected) regressions in a major version upgrade
> > are.
> > > > I've seen the community do something like this before (e.g. the major
> > > > rocksdb version bump in 1.14?) and it was quite valuable to know that
> > > > upfront!
> > > >
> > > > Best,
> > > > Mason
> > > >
> > > > On Fri, Oct 28, 2022 at 1:46 AM weijie guo <
> guoweijieres...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Yanfei for driving this.
> > > > >
> > > > > It allows us to easily find the problem of performance regression.
> > > > > Especially recently, I have made some improvements to the
> scheduling
> > > > > related parts, your work is very important to ensure that these
> > changes
> > > > do
> > > > > not cause some unexpected problems.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > > > >
> > > > > > Thanks for driving this and making the performance monitoring
> > public,
> > > > > this
> > > > > > can make us know and resolve the performance problem quickly.
> > > > > >
> > > > > > Looking forward to the workflow and detailed descriptions fo
> > > > > > flink-dev-benchmarks.
> > > > > >
> > > > > > Best,
> > > > > > Congxian
> > > > > >
> > > > > >
> > > > > > Yun Tang  于2022年10月27日周四 12:41写道：
> > > > > >
> > > > > > > Thanks, Yanfei for driving this to monitor the performance in
> the
> > > > > Apache
> > > > > > > Flink Slack Channel.
> > > > > > >
> > > > > > > Look forward to the workflow and detailed descriptions of
> > > > > > > flink-dev-benchmarks.
> > > > > > >
> > > > > > > Best
> > > > > > > Yun Tang
> > > > > > > 
> > > > > > > From: Hangxiang Yu 
> > > > > > > Sent: Thursday, October 27, 2022 10:59
> > > > > > > To: dev@flink.apache.org 
> > > > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from
> > > > > Ververica
> > > > > > > to Apache Flink Slack Channel
> > > > > > >
> > > > > > > Hi, Yanfei.
> > > > > > > Thanks for driving this.
> > > > > > > It could help us to detect and resolve the regression problem
> > > quickly
> > > > > and
> > > > > > > officially.
> > > > > > > I'd like to join as a maintainer.
> > > > > > > Looking forward to the workflow.
> > > > > > >
> > > > > > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei <
> yuanmei.w...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks, Yanfei, to drive this and make the performance
> > monitoring
> > > > > > > publicly
> > > > > > > > available.
> > > > > > > >
> > > > > > > > Looking forward to seeing the workflow, and more details as
> > > Martijn
> > 

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yanfei Lei]

 Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > > >
> > > > > Thanks for driving this and making the performance monitoring
> public,
> > > > this
> > > > > can make us know and resolve the performance problem quickly.
> > > > >
> > > > > Looking forward to the workflow and detailed descriptions fo
> > > > > flink-dev-benchmarks.
> > > > >
> > > > > Best,
> > > > > Congxian
> > > > >
> > > > >
> > > > > Yun Tang  于2022年10月27日周四 12:41写道：
> > > > >
> > > > > > Thanks, Yanfei for driving this to monitor the performance in the
> > > > Apache
> > > > > > Flink Slack Channel.
> > > > > >
> > > > > > Look forward to the workflow and detailed descriptions of
> > > > > > flink-dev-benchmarks.
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Hangxiang Yu 
> > > > > > Sent: Thursday, October 27, 2022 10:59
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from
> > > > Ververica
> > > > > > to Apache Flink Slack Channel
> > > > > >
> > > > > > Hi, Yanfei.
> > > > > > Thanks for driving this.
> > > > > > It could help us to detect and resolve the regression problem
> > quickly
> > > > and
> > > > > > officially.
> > > > > > I'd like to join as a maintainer.
> > > > > > Looking forward to the workflow.
> > > > > >
> > > > > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  >
> > > > wrote:
> > > > > >
> > > > > > > Thanks, Yanfei, to drive this and make the performance
> monitoring
> > > > > > publicly
> > > > > > > available.
> > > > > > >
> > > > > > > Looking forward to seeing the workflow, and more details as
> > Martijn
> > > > > > > mentioned.
> > > > > > >
> > > > > > > Best
> > > > > > > Yuan
> > > > > > >
> > > > > > > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser <
> > > > > martijnvis...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yanfei Lei,
> > > > > > > >
> > > > > > > > Thanks for setting this up! It would be interesting to also
> > know
> > > > > which
> > > > > > > > aspects of Flink are monitored for "performance". I'm
> assuming
> > > > there
> > > > > > are
> > > > > > > > specific pieces of functionality that are performance tested,
> > but
> > > > it
> > > > > > > would
> > > > > > > > be great if this would be written down somewhere (next to a
> > > > procedure
> > > > > > how
> > > > > > > > to detect a regression and what should be next steps).
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Martijn
> > > > > > > >
> > > > > > > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan <
> > > zakelly@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi yanfei,
> > > > > > > > >
> > > > > > > > > Thanks for driving this! It's a great help.
> > > > > > > > >
> > > > > > > > > I would like to join as a maintainer.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zakelly
> > > > > > > > >
> > > > > > > > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei <
> > > fredia...@gmail.com
> > > > >

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Martijn Visser]

Hi,

Is there any update to be expected on the benchmark? I see results of the
benchmark being posted to Slack, but it appears that it's not being
monitored and no follow-up actions are being taken. I think it's currently
lacking a process on how to interpret the results and what action should
be taken and by whom.

Best regards,

Martijn

On Thu, Nov 3, 2022 at 12:22 PM Jing Ge  wrote:

> Thanks yanfei for driving this!
>
> Looking forward to further discussion w.r.t. the workflow.
>
> Best regards,
> Jing
>
> On Mon, Oct 31, 2022 at 6:04 PM Mason Chen  wrote:
>
> > +1, thanks for driving this!
> >
> > On a side note, can we also ensure that a performance summary report for
> > Flink major version upgrades is in release notes, once this
> infrastructure
> > becomes mature? From the user perspective, it would be nice to know what
> > the expected (or unexpected) regressions in a major version upgrade are.
> > I've seen the community do something like this before (e.g. the major
> > rocksdb version bump in 1.14?) and it was quite valuable to know that
> > upfront!
> >
> > Best,
> > Mason
> >
> > On Fri, Oct 28, 2022 at 1:46 AM weijie guo 
> > wrote:
> >
> > > Thanks Yanfei for driving this.
> > >
> > > It allows us to easily find the problem of performance regression.
> > > Especially recently, I have made some improvements to the scheduling
> > > related parts, your work is very important to ensure that these changes
> > do
> > > not cause some unexpected problems.
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Congxian Qiu  于2022年10月28日周五 16:03写道：
> > >
> > > > Thanks for driving this and making the performance monitoring public,
> > > this
> > > > can make us know and resolve the performance problem quickly.
> > > >
> > > > Looking forward to the workflow and detailed descriptions fo
> > > > flink-dev-benchmarks.
> > > >
> > > > Best,
> > > > Congxian
> > > >
> > > >
> > > > Yun Tang  于2022年10月27日周四 12:41写道：
> > > >
> > > > > Thanks, Yanfei for driving this to monitor the performance in the
> > > Apache
> > > > > Flink Slack Channel.
> > > > >
> > > > > Look forward to the workflow and detailed descriptions of
> > > > > flink-dev-benchmarks.
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Hangxiang Yu 
> > > > > Sent: Thursday, October 27, 2022 10:59
> > > > > To: dev@flink.apache.org 
> > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from
> > > Ververica
> > > > > to Apache Flink Slack Channel
> > > > >
> > > > > Hi, Yanfei.
> > > > > Thanks for driving this.
> > > > > It could help us to detect and resolve the regression problem
> quickly
> > > and
> > > > > officially.
> > > > > I'd like to join as a maintainer.
> > > > > Looking forward to the workflow.
> > > > >
> > > > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei 
> > > wrote:
> > > > >
> > > > > > Thanks, Yanfei, to drive this and make the performance monitoring
> > > > > publicly
> > > > > > available.
> > > > > >
> > > > > > Looking forward to seeing the workflow, and more details as
> Martijn
> > > > > > mentioned.
> > > > > >
> > > > > > Best
> > > > > > Yuan
> > > > > >
> > > > > > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser <
> > > > martijnvis...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Yanfei Lei,
> > > > > > >
> > > > > > > Thanks for setting this up! It would be interesting to also
> know
> > > > which
> > > > > > > aspects of Flink are monitored for "performance". I'm assuming
> > > there
> > > > > are
> > > > > > > specific pieces of functionality that are performance tested,
> but
> > > it
> > > > > > would
> > > > > > > be great if this would be written do

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Jing Ge]

Thanks yanfei for driving this!

Looking forward to further discussion w.r.t. the workflow.

Best regards,
Jing

On Mon, Oct 31, 2022 at 6:04 PM Mason Chen  wrote:

> +1, thanks for driving this!
>
> On a side note, can we also ensure that a performance summary report for
> Flink major version upgrades is in release notes, once this infrastructure
> becomes mature? From the user perspective, it would be nice to know what
> the expected (or unexpected) regressions in a major version upgrade are.
> I've seen the community do something like this before (e.g. the major
> rocksdb version bump in 1.14?) and it was quite valuable to know that
> upfront!
>
> Best,
> Mason
>
> On Fri, Oct 28, 2022 at 1:46 AM weijie guo 
> wrote:
>
> > Thanks Yanfei for driving this.
> >
> > It allows us to easily find the problem of performance regression.
> > Especially recently, I have made some improvements to the scheduling
> > related parts, your work is very important to ensure that these changes
> do
> > not cause some unexpected problems.
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Congxian Qiu  于2022年10月28日周五 16:03写道：
> >
> > > Thanks for driving this and making the performance monitoring public,
> > this
> > > can make us know and resolve the performance problem quickly.
> > >
> > > Looking forward to the workflow and detailed descriptions fo
> > > flink-dev-benchmarks.
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Yun Tang  于2022年10月27日周四 12:41写道：
> > >
> > > > Thanks, Yanfei for driving this to monitor the performance in the
> > Apache
> > > > Flink Slack Channel.
> > > >
> > > > Look forward to the workflow and detailed descriptions of
> > > > flink-dev-benchmarks.
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Hangxiang Yu 
> > > > Sent: Thursday, October 27, 2022 10:59
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from
> > Ververica
> > > > to Apache Flink Slack Channel
> > > >
> > > > Hi, Yanfei.
> > > > Thanks for driving this.
> > > > It could help us to detect and resolve the regression problem quickly
> > and
> > > > officially.
> > > > I'd like to join as a maintainer.
> > > > Looking forward to the workflow.
> > > >
> > > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei 
> > wrote:
> > > >
> > > > > Thanks, Yanfei, to drive this and make the performance monitoring
> > > > publicly
> > > > > available.
> > > > >
> > > > > Looking forward to seeing the workflow, and more details as Martijn
> > > > > mentioned.
> > > > >
> > > > > Best
> > > > > Yuan
> > > > >
> > > > > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser <
> > > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Yanfei Lei,
> > > > > >
> > > > > > Thanks for setting this up! It would be interesting to also know
> > > which
> > > > > > aspects of Flink are monitored for "performance". I'm assuming
> > there
> > > > are
> > > > > > specific pieces of functionality that are performance tested, but
> > it
> > > > > would
> > > > > > be great if this would be written down somewhere (next to a
> > procedure
> > > > how
> > > > > > to detect a regression and what should be next steps).
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan <
> zakelly@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi yanfei,
> > > > > > >
> > > > > > > Thanks for driving this! It's a great help.
> > > > > > >
> > > > > > > I would like to join as a maintainer.
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > &g

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Mason Chen]

+1, thanks for driving this!

On a side note, can we also ensure that a performance summary report for
Flink major version upgrades is in release notes, once this infrastructure
becomes mature? From the user perspective, it would be nice to know what
the expected (or unexpected) regressions in a major version upgrade are.
I've seen the community do something like this before (e.g. the major
rocksdb version bump in 1.14?) and it was quite valuable to know that
upfront!

Best,
Mason

On Fri, Oct 28, 2022 at 1:46 AM weijie guo 
wrote:

> Thanks Yanfei for driving this.
>
> It allows us to easily find the problem of performance regression.
> Especially recently, I have made some improvements to the scheduling
> related parts, your work is very important to ensure that these changes do
> not cause some unexpected problems.
>
> Best regards,
>
> Weijie
>
>
> Congxian Qiu  于2022年10月28日周五 16:03写道：
>
> > Thanks for driving this and making the performance monitoring public,
> this
> > can make us know and resolve the performance problem quickly.
> >
> > Looking forward to the workflow and detailed descriptions fo
> > flink-dev-benchmarks.
> >
> > Best,
> > Congxian
> >
> >
> > Yun Tang  于2022年10月27日周四 12:41写道：
> >
> > > Thanks, Yanfei for driving this to monitor the performance in the
> Apache
> > > Flink Slack Channel.
> > >
> > > Look forward to the workflow and detailed descriptions of
> > > flink-dev-benchmarks.
> > >
> > > Best
> > > Yun Tang
> > > ________
> > > From: Hangxiang Yu 
> > > Sent: Thursday, October 27, 2022 10:59
> > > To: dev@flink.apache.org 
> > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from
> Ververica
> > > to Apache Flink Slack Channel
> > >
> > > Hi, Yanfei.
> > > Thanks for driving this.
> > > It could help us to detect and resolve the regression problem quickly
> and
> > > officially.
> > > I'd like to join as a maintainer.
> > > Looking forward to the workflow.
> > >
> > > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei 
> wrote:
> > >
> > > > Thanks, Yanfei, to drive this and make the performance monitoring
> > > publicly
> > > > available.
> > > >
> > > > Looking forward to seeing the workflow, and more details as Martijn
> > > > mentioned.
> > > >
> > > > Best
> > > > Yuan
> > > >
> > > > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Yanfei Lei,
> > > > >
> > > > > Thanks for setting this up! It would be interesting to also know
> > which
> > > > > aspects of Flink are monitored for "performance". I'm assuming
> there
> > > are
> > > > > specific pieces of functionality that are performance tested, but
> it
> > > > would
> > > > > be great if this would be written down somewhere (next to a
> procedure
> > > how
> > > > > to detect a regression and what should be next steps).
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan  >
> > > > wrote:
> > > > >
> > > > > > Hi yanfei,
> > > > > >
> > > > > > Thanks for driving this! It's a great help.
> > > > > >
> > > > > > I would like to join as a maintainer.
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei  >
> > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > As discussed earlier, we plan to create a benchmark channel in
> > > Apache
> > > > > > Flink
> > > > > > > slack[1], but the plan was shelved for a while[2]. So I went on
> > > with
> > > > > this
> > > > > > > work, and created the #flink-dev-benchmarks channel for
> > performance
> > > > > > > regression notifications.
> > > > > > >
> > > > > > > We have a regression report sc

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[weijie guo]

Thanks Yanfei for driving this.

It allows us to easily find the problem of performance regression.
Especially recently, I have made some improvements to the scheduling
related parts, your work is very important to ensure that these changes do
not cause some unexpected problems.

Best regards,

Weijie


Congxian Qiu  于2022年10月28日周五 16:03写道：

> Thanks for driving this and making the performance monitoring public,  this
> can make us know and resolve the performance problem quickly.
>
> Looking forward to the workflow and detailed descriptions fo
> flink-dev-benchmarks.
>
> Best,
> Congxian
>
>
> Yun Tang  于2022年10月27日周四 12:41写道：
>
> > Thanks, Yanfei for driving this to monitor the performance in the Apache
> > Flink Slack Channel.
> >
> > Look forward to the workflow and detailed descriptions of
> > flink-dev-benchmarks.
> >
> > Best
> > Yun Tang
> > 
> > From: Hangxiang Yu 
> > Sent: Thursday, October 27, 2022 10:59
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica
> > to Apache Flink Slack Channel
> >
> > Hi, Yanfei.
> > Thanks for driving this.
> > It could help us to detect and resolve the regression problem quickly and
> > officially.
> > I'd like to join as a maintainer.
> > Looking forward to the workflow.
> >
> > On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:
> >
> > > Thanks, Yanfei, to drive this and make the performance monitoring
> > publicly
> > > available.
> > >
> > > Looking forward to seeing the workflow, and more details as Martijn
> > > mentioned.
> > >
> > > Best
> > > Yuan
> > >
> > > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Yanfei Lei,
> > > >
> > > > Thanks for setting this up! It would be interesting to also know
> which
> > > > aspects of Flink are monitored for "performance". I'm assuming there
> > are
> > > > specific pieces of functionality that are performance tested, but it
> > > would
> > > > be great if this would be written down somewhere (next to a procedure
> > how
> > > > to detect a regression and what should be next steps).
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> > > wrote:
> > > >
> > > > > Hi yanfei,
> > > > >
> > > > > Thanks for driving this! It's a great help.
> > > > >
> > > > > I would like to join as a maintainer.
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> > > wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > As discussed earlier, we plan to create a benchmark channel in
> > Apache
> > > > > Flink
> > > > > > slack[1], but the plan was shelved for a while[2]. So I went on
> > with
> > > > this
> > > > > > work, and created the #flink-dev-benchmarks channel for
> performance
> > > > > > regression notifications.
> > > > > >
> > > > > > We have a regression report script[3] that runs daily, and a
> > > > notification
> > > > > > would be sent to the slack channel when the last few benchmark
> > > results
> > > > > are
> > > > > > significantly worse than the baseline.
> > > > > > Note, regressions are detected by a simple script which may have
> > > false
> > > > > > positives and false negatives. And all benchmarks are executed on
> > one
> > > > > > physical machine[4] which is provided by Ververica(Alibaba)[5],
> it
> > > > might
> > > > > > happen that hardware issues affect performance, like
> "[FLINK-18614
> > > > > > <https://issues.apache.org/jira/browse/FLINK-18614>] Performance
> > > > > regression
> > > > > > 2020.07.13"[6].
> > > > > >
> > > > > > After the migration, we need a procedure to watch over the entire
> > > > > > performance of Flink code together. F

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Congxian Qiu]

Thanks for driving this and making the performance monitoring public,  this
can make us know and resolve the performance problem quickly.

Looking forward to the workflow and detailed descriptions fo
flink-dev-benchmarks.

Best,
Congxian


Yun Tang  于2022年10月27日周四 12:41写道：

> Thanks, Yanfei for driving this to monitor the performance in the Apache
> Flink Slack Channel.
>
> Look forward to the workflow and detailed descriptions of
> flink-dev-benchmarks.
>
> Best
> Yun Tang
> 
> From: Hangxiang Yu 
> Sent: Thursday, October 27, 2022 10:59
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica
> to Apache Flink Slack Channel
>
> Hi, Yanfei.
> Thanks for driving this.
> It could help us to detect and resolve the regression problem quickly and
> officially.
> I'd like to join as a maintainer.
> Looking forward to the workflow.
>
> On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:
>
> > Thanks, Yanfei, to drive this and make the performance monitoring
> publicly
> > available.
> >
> > Looking forward to seeing the workflow, and more details as Martijn
> > mentioned.
> >
> > Best
> > Yuan
> >
> > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Yanfei Lei,
> > >
> > > Thanks for setting this up! It would be interesting to also know which
> > > aspects of Flink are monitored for "performance". I'm assuming there
> are
> > > specific pieces of functionality that are performance tested, but it
> > would
> > > be great if this would be written down somewhere (next to a procedure
> how
> > > to detect a regression and what should be next steps).
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> > wrote:
> > >
> > > > Hi yanfei,
> > > >
> > > > Thanks for driving this! It's a great help.
> > > >
> > > > I would like to join as a maintainer.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > As discussed earlier, we plan to create a benchmark channel in
> Apache
> > > > Flink
> > > > > slack[1], but the plan was shelved for a while[2]. So I went on
> with
> > > this
> > > > > work, and created the #flink-dev-benchmarks channel for performance
> > > > > regression notifications.
> > > > >
> > > > > We have a regression report script[3] that runs daily, and a
> > > notification
> > > > > would be sent to the slack channel when the last few benchmark
> > results
> > > > are
> > > > > significantly worse than the baseline.
> > > > > Note, regressions are detected by a simple script which may have
> > false
> > > > > positives and false negatives. And all benchmarks are executed on
> one
> > > > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> > > might
> > > > > happen that hardware issues affect performance, like "[FLINK-18614
> > > > > <https://issues.apache.org/jira/browse/FLINK-18614>] Performance
> > > > regression
> > > > > 2020.07.13"[6].
> > > > >
> > > > > After the migration, we need a procedure to watch over the entire
> > > > > performance of Flink code together. For example, if a regression
> > > > > occurs, investigating the cause and resolving the problem are
> needed.
> > > In
> > > > > the past, this procedure is maintained internally within Ververica,
> > but
> > > > we
> > > > > think making the procedure public would benefit all. I volunteer to
> > > serve
> > > > > as one of the initial maintainers, and would be glad if more
> > > contributors
> > > > > can join me. I'd also prepare some guidelines to help others get
> > > familiar
> > > > > with the workflow. I will start a new thread to discuss the
> workflow
> > > > soon.
> > > > >
> > > > >
> > > > > [1]
> https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > > > [3]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > > > [4] http://codespeed.dak8s.net:8080
> > > > > [5]
> https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > > > >
> > > > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> > > >
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yun Tang]

Thanks, Yanfei for driving this to monitor the performance in the Apache Flink 
Slack Channel.

Look forward to the workflow and detailed descriptions of flink-dev-benchmarks.

Best
Yun Tang

From: Hangxiang Yu 
Sent: Thursday, October 27, 2022 10:59
To: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to 
Apache Flink Slack Channel

Hi, Yanfei.
Thanks for driving this.
It could help us to detect and resolve the regression problem quickly and
officially.
I'd like to join as a maintainer.
Looking forward to the workflow.

On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:

> Thanks, Yanfei, to drive this and make the performance monitoring publicly
> available.
>
> Looking forward to seeing the workflow, and more details as Martijn
> mentioned.
>
> Best
> Yuan
>
> On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser 
> wrote:
>
> > Hi Yanfei Lei,
> >
> > Thanks for setting this up! It would be interesting to also know which
> > aspects of Flink are monitored for "performance". I'm assuming there are
> > specific pieces of functionality that are performance tested, but it
> would
> > be great if this would be written down somewhere (next to a procedure how
> > to detect a regression and what should be next steps).
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> wrote:
> >
> > > Hi yanfei,
> > >
> > > Thanks for driving this! It's a great help.
> > >
> > > I would like to join as a maintainer.
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > As discussed earlier, we plan to create a benchmark channel in Apache
> > > Flink
> > > > slack[1], but the plan was shelved for a while[2]. So I went on with
> > this
> > > > work, and created the #flink-dev-benchmarks channel for performance
> > > > regression notifications.
> > > >
> > > > We have a regression report script[3] that runs daily, and a
> > notification
> > > > would be sent to the slack channel when the last few benchmark
> results
> > > are
> > > > significantly worse than the baseline.
> > > > Note, regressions are detected by a simple script which may have
> false
> > > > positives and false negatives. And all benchmarks are executed on one
> > > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> > might
> > > > happen that hardware issues affect performance, like "[FLINK-18614
> > > > <https://issues.apache.org/jira/browse/FLINK-18614>] Performance
> > > regression
> > > > 2020.07.13"[6].
> > > >
> > > > After the migration, we need a procedure to watch over the entire
> > > > performance of Flink code together. For example, if a regression
> > > > occurs, investigating the cause and resolving the problem are needed.
> > In
> > > > the past, this procedure is maintained internally within Ververica,
> but
> > > we
> > > > think making the procedure public would benefit all. I volunteer to
> > serve
> > > > as one of the initial maintainers, and would be glad if more
> > contributors
> > > > can join me. I'd also prepare some guidelines to help others get
> > familiar
> > > > with the workflow. I will start a new thread to discuss the workflow
> > > soon.
> > > >
> > > >
> > > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > > [3]
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > > [4] http://codespeed.dak8s.net:8080
> > > > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > > >
> > > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> > >
> >
>


--
Best,
Hangxiang.

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Hangxiang Yu]

Hi, Yanfei.
Thanks for driving this.
It could help us to detect and resolve the regression problem quickly and
officially.
I'd like to join as a maintainer.
Looking forward to the workflow.

On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:

> Thanks, Yanfei, to drive this and make the performance monitoring publicly
> available.
>
> Looking forward to seeing the workflow, and more details as Martijn
> mentioned.
>
> Best
> Yuan
>
> On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser 
> wrote:
>
> > Hi Yanfei Lei,
> >
> > Thanks for setting this up! It would be interesting to also know which
> > aspects of Flink are monitored for "performance". I'm assuming there are
> > specific pieces of functionality that are performance tested, but it
> would
> > be great if this would be written down somewhere (next to a procedure how
> > to detect a regression and what should be next steps).
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> wrote:
> >
> > > Hi yanfei,
> > >
> > > Thanks for driving this! It's a great help.
> > >
> > > I would like to join as a maintainer.
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > As discussed earlier, we plan to create a benchmark channel in Apache
> > > Flink
> > > > slack[1], but the plan was shelved for a while[2]. So I went on with
> > this
> > > > work, and created the #flink-dev-benchmarks channel for performance
> > > > regression notifications.
> > > >
> > > > We have a regression report script[3] that runs daily, and a
> > notification
> > > > would be sent to the slack channel when the last few benchmark
> results
> > > are
> > > > significantly worse than the baseline.
> > > > Note, regressions are detected by a simple script which may have
> false
> > > > positives and false negatives. And all benchmarks are executed on one
> > > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> > might
> > > > happen that hardware issues affect performance, like "[FLINK-18614
> > > > ] Performance
> > > regression
> > > > 2020.07.13"[6].
> > > >
> > > > After the migration, we need a procedure to watch over the entire
> > > > performance of Flink code together. For example, if a regression
> > > > occurs, investigating the cause and resolving the problem are needed.
> > In
> > > > the past, this procedure is maintained internally within Ververica,
> but
> > > we
> > > > think making the procedure public would benefit all. I volunteer to
> > serve
> > > > as one of the initial maintainers, and would be glad if more
> > contributors
> > > > can join me. I'd also prepare some guidelines to help others get
> > familiar
> > > > with the workflow. I will start a new thread to discuss the workflow
> > > soon.
> > > >
> > > >
> > > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > > [3]
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > > [4] http://codespeed.dak8s.net:8080
> > > > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > > >
> > > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yuan Mei]

Thanks, Yanfei, to drive this and make the performance monitoring publicly
available.

Looking forward to seeing the workflow, and more details as Martijn
mentioned.

Best
Yuan

On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser 
wrote:

> Hi Yanfei Lei,
>
> Thanks for setting this up! It would be interesting to also know which
> aspects of Flink are monitored for "performance". I'm assuming there are
> specific pieces of functionality that are performance tested, but it would
> be great if this would be written down somewhere (next to a procedure how
> to detect a regression and what should be next steps).
>
> Best regards,
>
> Martijn
>
> On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan  wrote:
>
> > Hi yanfei,
> >
> > Thanks for driving this! It's a great help.
> >
> > I would like to join as a maintainer.
> >
> > Best,
> > Zakelly
> >
> > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei  wrote:
> > >
> > > Hi everyone,
> > >
> > > As discussed earlier, we plan to create a benchmark channel in Apache
> > Flink
> > > slack[1], but the plan was shelved for a while[2]. So I went on with
> this
> > > work, and created the #flink-dev-benchmarks channel for performance
> > > regression notifications.
> > >
> > > We have a regression report script[3] that runs daily, and a
> notification
> > > would be sent to the slack channel when the last few benchmark results
> > are
> > > significantly worse than the baseline.
> > > Note, regressions are detected by a simple script which may have false
> > > positives and false negatives. And all benchmarks are executed on one
> > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> might
> > > happen that hardware issues affect performance, like "[FLINK-18614
> > > ] Performance
> > regression
> > > 2020.07.13"[6].
> > >
> > > After the migration, we need a procedure to watch over the entire
> > > performance of Flink code together. For example, if a regression
> > > occurs, investigating the cause and resolving the problem are needed.
> In
> > > the past, this procedure is maintained internally within Ververica, but
> > we
> > > think making the procedure public would benefit all. I volunteer to
> serve
> > > as one of the initial maintainers, and would be glad if more
> contributors
> > > can join me. I'd also prepare some guidelines to help others get
> familiar
> > > with the workflow. I will start a new thread to discuss the workflow
> > soon.
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > [3]
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > [4] http://codespeed.dak8s.net:8080
> > > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > >
> > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> >
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Martijn Visser]

Hi Yanfei Lei,

Thanks for setting this up! It would be interesting to also know which
aspects of Flink are monitored for "performance". I'm assuming there are
specific pieces of functionality that are performance tested, but it would
be great if this would be written down somewhere (next to a procedure how
to detect a regression and what should be next steps).

Best regards,

Martijn

On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan  wrote:

> Hi yanfei,
>
> Thanks for driving this! It's a great help.
>
> I would like to join as a maintainer.
>
> Best,
> Zakelly
>
> On Wed, Oct 26, 2022 at 11:32 AM yanfei lei  wrote:
> >
> > Hi everyone,
> >
> > As discussed earlier, we plan to create a benchmark channel in Apache
> Flink
> > slack[1], but the plan was shelved for a while[2]. So I went on with this
> > work, and created the #flink-dev-benchmarks channel for performance
> > regression notifications.
> >
> > We have a regression report script[3] that runs daily, and a notification
> > would be sent to the slack channel when the last few benchmark results
> are
> > significantly worse than the baseline.
> > Note, regressions are detected by a simple script which may have false
> > positives and false negatives. And all benchmarks are executed on one
> > physical machine[4] which is provided by Ververica(Alibaba)[5], it might
> > happen that hardware issues affect performance, like "[FLINK-18614
> > ] Performance
> regression
> > 2020.07.13"[6].
> >
> > After the migration, we need a procedure to watch over the entire
> > performance of Flink code together. For example, if a regression
> > occurs, investigating the cause and resolving the problem are needed. In
> > the past, this procedure is maintained internally within Ververica, but
> we
> > think making the procedure public would benefit all. I volunteer to serve
> > as one of the initial maintainers, and would be glad if more contributors
> > can join me. I'd also prepare some guidelines to help others get familiar
> > with the workflow. I will start a new thread to discuss the workflow
> soon.
> >
> >
> > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > [3]
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > [4] http://codespeed.dak8s.net:8080
> > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> >
> > [6] https://issues.apache.org/jira/browse/FLINK-18614
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Zakelly Lan]

Hi yanfei,

Thanks for driving this! It's a great help.

I would like to join as a maintainer.

Best,
Zakelly

On Wed, Oct 26, 2022 at 11:32 AM yanfei lei  wrote:
>
> Hi everyone,
>
> As discussed earlier, we plan to create a benchmark channel in Apache Flink
> slack[1], but the plan was shelved for a while[2]. So I went on with this
> work, and created the #flink-dev-benchmarks channel for performance
> regression notifications.
>
> We have a regression report script[3] that runs daily, and a notification
> would be sent to the slack channel when the last few benchmark results are
> significantly worse than the baseline.
> Note, regressions are detected by a simple script which may have false
> positives and false negatives. And all benchmarks are executed on one
> physical machine[4] which is provided by Ververica(Alibaba)[5], it might
> happen that hardware issues affect performance, like "[FLINK-18614
> ] Performance regression
> 2020.07.13"[6].
>
> After the migration, we need a procedure to watch over the entire
> performance of Flink code together. For example, if a regression
> occurs, investigating the cause and resolving the problem are needed. In
> the past, this procedure is maintained internally within Ververica, but we
> think making the procedure public would benefit all. I volunteer to serve
> as one of the initial maintainers, and would be glad if more contributors
> can join me. I'd also prepare some guidelines to help others get familiar
> with the workflow. I will start a new thread to discuss the workflow soon.
>
>
> [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> [2] https://issues.apache.org/jira/browse/FLINK-28468
> [3]
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> [4] http://codespeed.dak8s.net:8080
> [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
>
> [6] https://issues.apache.org/jira/browse/FLINK-18614

[ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[yanfei lei]

Hi everyone,

As discussed earlier, we plan to create a benchmark channel in Apache Flink
slack[1], but the plan was shelved for a while[2]. So I went on with this
work, and created the #flink-dev-benchmarks channel for performance
regression notifications.

We have a regression report script[3] that runs daily, and a notification
would be sent to the slack channel when the last few benchmark results are
significantly worse than the baseline.
Note, regressions are detected by a simple script which may have false
positives and false negatives. And all benchmarks are executed on one
physical machine[4] which is provided by Ververica(Alibaba)[5], it might
happen that hardware issues affect performance, like "[FLINK-18614
] Performance regression
2020.07.13"[6].

After the migration, we need a procedure to watch over the entire
performance of Flink code together. For example, if a regression
occurs, investigating the cause and resolving the problem are needed. In
the past, this procedure is maintained internally within Ververica, but we
think making the procedure public would benefit all. I volunteer to serve
as one of the initial maintainers, and would be glad if more contributors
can join me. I'd also prepare some guidelines to help others get familiar
with the workflow. I will start a new thread to discuss the workflow soon.


[1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
[2] https://issues.apache.org/jira/browse/FLINK-28468
[3]
https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
[4] http://codespeed.dak8s.net:8080
[5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro

[6] https://issues.apache.org/jira/browse/FLINK-18614