> However, I will point out that this change is not the same as the
> initially proposed change to increase the number of times a new test
> is run (point (1) in the original email for this thread). I am not
that's true. Increasing the number of times to run a test method is executed
could be a b
Good question. I agree that we should strive to fix flaky tests as top priority.
For more background, please read the document that was compiled about a year
ago:
https://docs.google.com/document/d/10lmn4pW1IsT_8D1ZE0vMjASX0HhjdGdjB794iyScwns/edit
and these mailing list discussions:
https://list
Thank your reply. I agree with the idea mentioned above.
However, I have a question: Why don't we fix all the flaky tests first? After,
We can not allow all tests to retry.
1. Fix all existing flaky tests.
2. Then, in each PR, for all tests, as long as they fail, let the job fail
3. If there a
> Alternatively, would it be possible to only retry tests that are
annotated as "flaky"? This would exclude all new tests, by default,
and it'd make it clearer which tests need to be fixed.
+1,
If a test in other group which we can confirm it is a flaky test,
we can move it to flaky group.
Pengh
This is a great discussion, thanks for starting it.
I support turning off retries for new tests, as Lari suggests.
However, I will point out that this change is not the same as the
initially proposed change to increase the number of times a new test
is run (point (1) in the original email for this
Good job!
I see. Now we retry all tests by default. What's the reason?
The process I understand is:
We don't need to retry the unit test. Once the test fails, let the job fail.
The difference is that new or modified tests need to be tested many times. As
long as there is one failure, the job wi
Here's a PR to disable tests retries for new and modified tests:
https://github.com/apache/pulsar/pull/14607
Please review and provide feedback.
-Lari
On 2022/03/08 11:24:13 Lari Hotari wrote:
> I think this is a good idea. In CI, the tests will be retried multiple
> times at multiple levels: at
I think this is a good idea. In CI, the tests will be retried multiple
times at multiple levels: at the test class / test method level and at the
build level.
I'd like to get rid of test retries completely, since that is the reason
why the flakiness problem gets worse over time. This was also cove
Il giorno mar 8 mar 2022 alle ore 03:08 PengHui Li
ha scritto:
>
> I think we can try this idea, it looks like to find a way to
> ensure the new changes from the test will not introduce
> new uncertainty.
Yes, "trying" is good.
+1
Enrico
>
> Penghui
>
> On Mon, Mar 7, 2022 at 11:27 PM Baozi
>
I think we can try this idea, it looks like to find a way to
ensure the new changes from the test will not introduce
new uncertainty.
Penghui
On Mon, Mar 7, 2022 at 11:27 PM Baozi
wrote:
> Thank for your reply. I have some different viewpoint. Please have a look.
>
> > I am not sure that this h
Thank for your reply. I have some different viewpoint. Please have a look.
> I am not sure that this helps.
> You can still be very lucky and unfortunately many times the impact is on
> other tests, not the new tests.
Yes, that cannot avoid other unit tests affected by this PR. However, the
stab
Il Lun 7 Mar 2022, 07:46 Haiting Jiang ha scritto:
> +1 for this great idea.
> Although I am not sure if there is an easy and accurate way to "find new
> or modified unit tests".
>
> Thanks,
> Haiting
>
> On 2022/03/07 03:11:47 包子 wrote:
> > Hi, I want to discuss how to improve the stability of u
+1 for this great idea.
Although I am not sure if there is an easy and accurate way to "find new or
modified unit tests".
Thanks,
Haiting
On 2022/03/07 03:11:47 包子 wrote:
> Hi, I want to discuss how to improve the stability of unit testing.I found
> that most flaky unit tests can be reproduc lo
Hi, I want to discuss how to improve the stability of unit testing.I found that
most flaky unit tests can be reproduc locally and only need to be executed a
few more times.
Can we add the following mandatory constraints to ensure the stability of unit
testing of code?
1. If new / modified unit
14 matches
Mail list logo
