+1
I don't have anything substantive to add, but I want to express how pleased
I am to see this conversation happening.
David
On Thu, Feb 2, 2023 at 5:09 AM Martijn Visser
wrote:
> Hi all,
>
> +1 for the overall proposal. My feedback matches with what Matthias
> has already provided earlier.
>
Hi all,
+1 for the overall proposal. My feedback matches with what Matthias
has already provided earlier.
Best regards,
Martijn
Op di 31 jan. 2023 om 19:01 schreef Matthias Pohl
:
>
> Thanks for the effort you put into this discussion, Yanfei.
>
> For me, the regression tests are similar to tes
Thanks for the effort you put into this discussion, Yanfei.
For me, the regression tests are similar to test instabilities. In this
sense, I agree with what Piotr and Jing said about it:
- It needs to be identified and fixed as soon as possible to avoid the
change affecting other contributions (e.
Hi Yanfei,
Thanks for your proposal and effort of driving it.
I really like the document on how to deal with the performance regressions.
This will coach more developers to be able to work with it. I would suggest
that more developers will be aware of the performance regressions during
the daily
Hi,
@Dong, I think the drawback of your proposal is that it wouldn't detect if
there is visible performance regression within benchmark noise. While this
should be do-able with large enough number of samples
@Yanfei, Gathering medians would have basically the same problems with how
to deal with t
Thanks for all the feedback and suggestions.
@Piotr:
>> I was setting the priority to a blocker and I would propose to add this to
>> the instructions and general convention.
Thanks for sharing your experience, I will update this to the document.
And your suggestion of leveraging Two-sample Kolm
Hi Piotr,
Yes, the challenge of developing such an automatic tool is indeed to handle
noise and achieve a balance between false positive and false negative. It
is great to know that we already have scripts that can access historical
benchmark data and generate alerts.
There is some heuristic that
Hi Dong,
The main issue with an automatic tool at the moment is that some benchmarks
are quite noisy and performance regressions are often within the noise of a
given benchmark. Our currently existing tooling can not handle those cases.
Until we address this issue, I think it will have to remain a
Hi Yanfei,
Thanks for driving the benchmark monitoring effort! The Google doc and the
community wiki looks pretty good.
According to Yuan's comment, it seems that we currently manually watch the
benchmark results to detect regression. Have we considered automating this
process by e.g. exporting t
Hi,
Thanks for bringing this up! Generally speaking +1 for the proposal. I have
only one suggestion for the draft.
In the past years, when I was creating performance regression tickets, I
was setting the priority to a blocker and I would propose to add this to
the instructions and general convent
Hey Yanfei,
Thanks so much for the efforts driving the whole process. It's great to see
that the performance benchmarks are indeed useful to help find regressions.
This is a discussion thread separated from the original performance
benchmark announcement thread [1]. Let's continue here so that mor
Hi devs,
I'd like to start a discussion about incorporating performance
regression monitoring into the routine process. Flink benchmarks are
periodically executed on http://codespeed.dak8s.net:8080 to monitor
Flink performance. In late Oct'22, a new slack channel
#flink-dev-benchmarks was created
12 matches
Mail list logo
