enforcer-rules: standard vs. extra-enforcer-rules

2014-05-28 Thread Mirko Friedenhagen
Hello everybody,

there is an outstanding MENFORCER-193[0] request for a new standard
rule, which will allow to ban repositories. What is your opinion about
adding new standard rules in enforcer vs. adding to Mojo's
extra-enforcer-rules?

Regards Mirko
[0] https://jira.codehaus.org/browse/MENFORCER-193
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen)
https://bitbucket.org/mfriedenhagen/

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Kristian Rosenvold
I would really like to see and understand properly how this would
collaborate with JUnit rules for retries. I suspect there might be
some interesting issues regarding reporting (i.e. can the existing
logic handle reporting of n different executions of a single test).
How does this explode i parallel junit is selected :) ?

Technically you can just pass a system property from your pom and into
the surefire fork with the number of retries, you can read this
proprety in the junit rule. I really think you should start with this
approach and flesh out the details from there. It might turn out this
is all you need, or that some other entirely unforseen issues pop up.

As for the actual property in the plugin, I really think we should see
where this leads before deciding on how to do this. On a similar note,
I know some people with large corporate clouds sometimes run a test on
1 different nodes in parallel, to investigate if there is any kind
of 1/1 type failures, just to be able to summarize in 20 seconds
instead of 1 linear runs.

This might also be another one of those cases where we could consider
picking up user-written code from the test classpath; but I think I
need another year thinking about that :)

Kristian




2014-05-28 19:58 GMT+02:00 Jason van Zyl :
> I personally don't see an issue with those feature additions, but Kristian 
> does the lions share of the work on the Surefire plugin so it would be his 
> call. I would definitely support the additions of those features and would 
> help test.
>
> On May 28, 2014, at 1:46 PM, Qingzhou Luo  wrote:
>
>> Hi Jason,
>>
>> Thanks for your reply. We plan to first extend surefire, so users can put a 
>> boolean parameter rerunFailintTests and an int parameter 
>> rerunFailintTestsCount, to tell surefire whether to rerun failing tests 
>> immediately after they fail. If a test passes in any of those reruns if will 
>> be marked as pass, but all the failing attempts will still be recorded and 
>> may be presented to users in a different way.
>>
>> We plan to finish everything in next 1-2 months. So we wonder would you be 
>> willing to accept our contribution in the end, and patch our code into 
>> surefire?
>>
>> Thanks,
>>
>> Qingzhou
>>
>>
>> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl  wrote:
>>
>> On May 27, 2014, at 1:14 PM, Qingzhou Luo  wrote:
>>
>>> Hi,
>>>
>>> I am an intern at Google. The first step of my internship project is to add
>>> the ability to Maven to automatically rerun failing tests a few times, to
>>> see if they ever pass in any of the reruns. It is useful because in many
>>> cases a test fails because it is flaky, not because there is a bug in the
>>> new source code change.
>>>
>>
>> I think generally being able to re-run failed tests is useful. I use this 
>> feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're 
>> trying to identify flaky tests I think that's useful especially if you can 
>> collect statistics. While In the long term you may not want to rely on the 
>> re-running of failed tests that later succeed without alteration as the 
>> basis for good testing, but it's a good identification tool, but a useful 
>> feature in its own right.
>>
>>> We think the right way to achieve this is to modify surefire plugin of
>>> maven. We want to add it as a part of the configuration of surefire, so
>>> users can decide whether they want to enable this feature, and how many
>>> times they want to rerun failing tests. We plan to open-source our
>>> contribution, and hopefully can merge our code into surefire master branch
>>> in the end. Therefore,  we are wondering do you have any
>>> comments/suggestions/opinions regarding this? We appreciate any input.
>>>
>>
>> Maybe a simple mechanism where you store the failed tests in a file and then 
>> have a flag to run only the failed tests. I think this would be very useful. 
>> Then possible a small, configurable loop around the main execution of tests 
>> if you wanted to specify how many times to run the tests.
>>
>>> Thank you very much!
>>>
>>> Regards,
>>>
>>> Qingzhou
>>
>> Thanks,
>>
>> Jason
>>
>> --
>> Jason van Zyl
>> Founder,  Apache Maven
>> http://twitter.com/jvanzyl
>> http://twitter.com/takari_io
>> -
>>
>> The modern conservative is engaged in one of man's oldest exercises in moral 
>> philosophy; that is,
>> the search for a superior moral justification for selfishness.
>>
>>  -- John Kenneth Galbraith
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> Thanks,
>
> Jason
>
> --
> Jason van Zyl
> Founder,  Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> -
>
> Selfish deeds are the shortest path to self destruction.
>
>  -- The Seven Samuari, Akira Kurosawa
>
>
>
>
>
>
>
>
>

---

Re: Extend surefire to rerun failing tests

2014-05-28 Thread Jason van Zyl
I personally don't see an issue with those feature additions, but Kristian does 
the lions share of the work on the Surefire plugin so it would be his call. I 
would definitely support the additions of those features and would help test.

On May 28, 2014, at 1:46 PM, Qingzhou Luo  wrote:

> Hi Jason,
> 
> Thanks for your reply. We plan to first extend surefire, so users can put a 
> boolean parameter rerunFailintTests and an int parameter 
> rerunFailintTestsCount, to tell surefire whether to rerun failing tests 
> immediately after they fail. If a test passes in any of those reruns if will 
> be marked as pass, but all the failing attempts will still be recorded and 
> may be presented to users in a different way.
> 
> We plan to finish everything in next 1-2 months. So we wonder would you be 
> willing to accept our contribution in the end, and patch our code into 
> surefire?
> 
> Thanks,
> 
> Qingzhou
> 
> 
> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl  wrote:
> 
> On May 27, 2014, at 1:14 PM, Qingzhou Luo  wrote:
> 
>> Hi,
>> 
>> I am an intern at Google. The first step of my internship project is to add
>> the ability to Maven to automatically rerun failing tests a few times, to
>> see if they ever pass in any of the reruns. It is useful because in many
>> cases a test fails because it is flaky, not because there is a bug in the
>> new source code change.
>> 
> 
> I think generally being able to re-run failed tests is useful. I use this 
> feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're 
> trying to identify flaky tests I think that's useful especially if you can 
> collect statistics. While In the long term you may not want to rely on the 
> re-running of failed tests that later succeed without alteration as the basis 
> for good testing, but it's a good identification tool, but a useful feature 
> in its own right.
> 
>> We think the right way to achieve this is to modify surefire plugin of
>> maven. We want to add it as a part of the configuration of surefire, so
>> users can decide whether they want to enable this feature, and how many
>> times they want to rerun failing tests. We plan to open-source our
>> contribution, and hopefully can merge our code into surefire master branch
>> in the end. Therefore,  we are wondering do you have any
>> comments/suggestions/opinions regarding this? We appreciate any input.
>> 
> 
> Maybe a simple mechanism where you store the failed tests in a file and then 
> have a flag to run only the failed tests. I think this would be very useful. 
> Then possible a small, configurable loop around the main execution of tests 
> if you wanted to specify how many times to run the tests.
> 
>> Thank you very much!
>> 
>> Regards,
>> 
>> Qingzhou
> 
> Thanks,
> 
> Jason
> 
> --
> Jason van Zyl
> Founder,  Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> -
> 
> The modern conservative is engaged in one of man's oldest exercises in moral 
> philosophy; that is, 
> the search for a superior moral justification for selfishness.
> 
>  -- John Kenneth Galbraith
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Thanks,

Jason

--
Jason van Zyl
Founder,  Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
-

Selfish deeds are the shortest path to self destruction.

 -- The Seven Samuari, Akira Kurosawa











Re: Extend surefire to rerun failing tests

2014-05-28 Thread Qingzhou Luo
Hi Jason,

Thanks for your reply. We plan to first extend surefire, so users can put a
boolean parameter rerunFailintTests and an int parameter
rerunFailintTestsCount, to tell surefire whether to rerun failing tests
immediately after they fail. If a test passes in any of those reruns if
will be marked as pass, but all the failing attempts will still be recorded
and may be presented to users in a different way.

We plan to finish everything in next 1-2 months. So we wonder would you be
willing to accept our contribution in the end, and patch our code into
surefire?

Thanks,

Qingzhou


On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl  wrote:

>
> On May 27, 2014, at 1:14 PM, Qingzhou Luo  wrote:
>
> Hi,
>
> I am an intern at Google. The first step of my internship project is to add
> the ability to Maven to automatically rerun failing tests a few times, to
> see if they ever pass in any of the reruns. It is useful because in many
> cases a test fails because it is flaky, not because there is a bug in the
> new source code change.
>
>
> I think generally being able to re-run failed tests is useful. I use this
> feature in the IDE all the time in a fail/fix/run/repeat scenario. If
> you're trying to identify flaky tests I think that's useful especially if
> you can collect statistics. While In the long term you may not want to rely
> on the re-running of failed tests that later succeed without alteration as
> the basis for good testing, but it's a good identification tool, but a
> useful feature in its own right.
>
> We think the right way to achieve this is to modify surefire plugin of
> maven. We want to add it as a part of the configuration of surefire, so
> users can decide whether they want to enable this feature, and how many
> times they want to rerun failing tests. We plan to open-source our
> contribution, and hopefully can merge our code into surefire master branch
> in the end. Therefore,  we are wondering do you have any
> comments/suggestions/opinions regarding this? We appreciate any input.
>
>
> Maybe a simple mechanism where you store the failed tests in a file and
> then have a flag to run only the failed tests. I think this would be very
> useful. Then possible a small, configurable loop around the main execution
> of tests if you wanted to specify how many times to run the tests.
>
> Thank you very much!
>
> Regards,
>
> Qingzhou
>
>
> Thanks,
>
> Jason
>
> --
> Jason van Zyl
> Founder,  Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> -
>
>
> The modern conservative is engaged in one of man's oldest exercises in moral 
> philosophy; that is,
> the search for a superior moral justification for selfishness.
>
>  -- John Kenneth Galbraith
>
>
>
>
>
>
>
>
>
>


[GitHub] maven-integration-testing pull request: Added guidance for setting...

2014-05-28 Thread markdingram
GitHub user markdingram opened a pull request:

https://github.com/apache/maven-integration-testing/pull/5

Added guidance for setting proxy if required



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markdingram/maven-integration-testing 
temp-run-patch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/maven-integration-testing/pull/5.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5


commit e322da10152baa7e9a4e19676e5a5ca1856bc448
Author: Mark Ingram 
Date:   2014-05-28T16:29:58Z

MNG-5639 a test for resolving import scope POMs in DependencyManagement

The new feature is allowing the repository URL to contain a property.

commit 7123c6b4020d7b3c8ada5ad1b700bac38328d52d
Author: Mark Ingram 
Date:   2014-05-28T16:36:31Z

Added guidance for setting proxy if required

commit c3228470a3873538ab55f77e8e8ce9f592eaae04
Author: Mark Ingram 
Date:   2014-05-28T16:39:54Z

Revert "MNG-5639 a test for resolving import scope POMs in 
DependencyManagement"

This reverts commit e322da10152baa7e9a4e19676e5a5ca1856bc448.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



[GitHub] maven-integration-testing pull request: MNG-5639 a test for resolv...

2014-05-28 Thread markdingram
GitHub user markdingram opened a pull request:

https://github.com/apache/maven-integration-testing/pull/4

MNG-5639 a test for resolving import scope POMs in DependencyManagement

The new feature is allowing the repository URL to contain a property.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markdingram/maven-integration-testing master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/maven-integration-testing/pull/4.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4


commit 58f0471de1852c424d09c9f1577e7d5dc4177f4d
Author: Mark Ingram 
Date:   2014-05-28T16:30:48Z

MNG-5639 a test for resolving import scope POMs in DependencyManagement

The new feature is allowing the repository URL to contain a property.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Kristian Rosenvold
2014-05-28 14:41 GMT+02:00 Jason van Zyl :

> Maybe a simple mechanism where you store the failed tests in a file and then 
> have a flag to run only the failed tests. I think this would be very useful. 
> Then possible a small, configurable loop around the main execution of tests 
> if you wanted to specify how many times to run the tests.

Surefire already does this with the runorder=failedfirst. Most of the
other behaviour is handled quite all by junit rules, but since we
already store the failure/success information, I'm sure there's room
for other options :)

Kristian

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Alexander Kriegisch

I have not tried, Kristian, but it looks promising. Both your link and the 
randomised testing one I have forwarded to my old Scrum team (I am no longer 
coaching them, thus unable to access their code base and try for myself).

Thank you. :-)
-- 
Alexander Kriegisch


> Am 28.05.2014 um 09:50 schrieb Kristian Rosenvold 
> :
> 
> Does this fit the bill for you ?
> 
> http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately
> 
> Kristian
> 
> 
> 2014-05-28 9:14 GMT+02:00 Alexander Kriegisch :
>> 
>> 
>> --
>> Alexander Kriegisch
>> http://scrum-master.de
>> 
>>> Am 28.05.2014 um 05:13 schrieb Benson Margulies :
>>> 
>>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo wrote:
>>> 
 I am an intern at Google. The first step of my internship project is to add
 the ability to Maven to automatically rerun failing tests a few times, to
 see if they ever pass in any of the reruns. It is useful because in many
 cases a test fails because it is flaky, not because there is a bug in the
 new source code change.
>>> 
>>> I entirely disagree. 99.% of programs are completely deterministic. If
>>> you feed them the same inputs, you get the same output.
>> 
>> Well, in my previous project (I am an Agile Coach) my teams were working on 
>> a distributed, multi-threaded system and around 10% of all unit tests and 
>> next to all integration tests were affected by the exception you mentioned:
>> 
>>> The exceptions are tests that use threads, and it's quite hard to do a good
>>> job writing them.
>> 
>> One can argue that several of the unit tests were not real unit tests 
>> because they mocked some, but not all of the infrastructure they needed for 
>> running. Anyway, they were running in Surefire (the integration tests in 
>> Failsafe), and some lf them were flaky.
>> 
>>> Just 'rerunning a few times' does not lead to useful
>>> information in my view.
>> 
>> Only the information that indeed they are flaky and a rough percentage of 
>> cases in which they fail, which would be rather dependend on the test 
>> environment though. Better than nothing information-wise. So far, so good.
>> 
>> Having said all that, I still agree to Benson that a failing test should 
>> just fail, period. I do not want developers to think, "it is only failing 
>> only in 5% of all runs, so it is fine / low priority to fix". OTOH I 
>> understand Qingzhou from a perspective of wishing to "test the test" or 
>> rather wishing to harden a flaky test into a stable one, either by fixing 
>> the test or the threading bug(s) it exposes. Benson's suggestion below looks 
>> promising though, so thanks for that.
>> 
>>> if you want random testing, there's a fine alternative at
>>> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
>>> maven plugin. It can already 'rerun your tests a few times' very easily.
>>> 
>>> I wonder what the more involved surefire maintainers think, but as a PMC
>>> member I'd be -1 on this proposal.
>>> 
 We think the right way to achieve this is to modify surefire plugin of
 maven. We want to add it as a part of the configuration of surefire, so
 users can decide whether they want to enable this feature, and how many
 times they want to rerun failing tests. We plan to open-source our
 contribution, and hopefully can merge our code into surefire master branch
 in the end. Therefore,  we are wondering do you have any
 comments/suggestions/opinions regarding this? We appreciate any input.
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
>> For additional commands, e-mail: dev-h...@maven.apache.org
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Jason van Zyl

On May 27, 2014, at 1:14 PM, Qingzhou Luo  wrote:

> Hi,
> 
> I am an intern at Google. The first step of my internship project is to add
> the ability to Maven to automatically rerun failing tests a few times, to
> see if they ever pass in any of the reruns. It is useful because in many
> cases a test fails because it is flaky, not because there is a bug in the
> new source code change.
> 

I think generally being able to re-run failed tests is useful. I use this 
feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're 
trying to identify flaky tests I think that's useful especially if you can 
collect statistics. While In the long term you may not want to rely on the 
re-running of failed tests that later succeed without alteration as the basis 
for good testing, but it's a good identification tool, but a useful feature in 
its own right.

> We think the right way to achieve this is to modify surefire plugin of
> maven. We want to add it as a part of the configuration of surefire, so
> users can decide whether they want to enable this feature, and how many
> times they want to rerun failing tests. We plan to open-source our
> contribution, and hopefully can merge our code into surefire master branch
> in the end. Therefore,  we are wondering do you have any
> comments/suggestions/opinions regarding this? We appreciate any input.
> 

Maybe a simple mechanism where you store the failed tests in a file and then 
have a flag to run only the failed tests. I think this would be very useful. 
Then possible a small, configurable loop around the main execution of tests if 
you wanted to specify how many times to run the tests.

> Thank you very much!
> 
> Regards,
> 
> Qingzhou

Thanks,

Jason

--
Jason van Zyl
Founder,  Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
-

The modern conservative is engaged in one of man's oldest exercises in moral 
philosophy; that is, 
the search for a superior moral justification for selfishness.

 -- John Kenneth Galbraith











[GitHub] maven-enforcer pull request: [MENFORCER-193]: Add new rule: Banned...

2014-05-28 Thread wangyf2010
GitHub user wangyf2010 opened a pull request:

https://github.com/apache/maven-enforcer/pull/13

[MENFORCER-193]: Add new rule: BannedRepositories to ban specified repos...

There are use cases that need to ban specified repositories.
Ex. one enterprise migrate their repositories from old one to new one.
But some users still use old settings.xml or some projects' pom.xml still 
have old repositories.

What this rule did:
1. bannedRepositories: user could add banned repositories and support 
wildcard "*" to simplify user's usage.
2. allowedRepositories: that's simpler and useful for enterprise users.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyf2010/maven-enforcer trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/maven-enforcer/pull/13.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13


commit e5d5dc91a51ca552b9da2f2dada0825fe04ffdc6
Author: Simon, Wang 
Date:   2014-05-28T10:06:33Z

[MENFORCER-193]: Add new rule: BannedRepositories to ban specified 
repositories for whole maven session




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Kristian Rosenvold
Does this fit the bill for you ?

http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately

Kristian


2014-05-28 9:14 GMT+02:00 Alexander Kriegisch :
>
>
> --
> Alexander Kriegisch
> http://scrum-master.de
>
> Am 28.05.2014 um 05:13 schrieb Benson Margulies :
>
>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo wrote:
>>
>>> I am an intern at Google. The first step of my internship project is to add
>>> the ability to Maven to automatically rerun failing tests a few times, to
>>> see if they ever pass in any of the reruns. It is useful because in many
>>> cases a test fails because it is flaky, not because there is a bug in the
>>> new source code change.
>>
>> I entirely disagree. 99.% of programs are completely deterministic. If
>> you feed them the same inputs, you get the same output.
>
> Well, in my previous project (I am an Agile Coach) my teams were working on a 
> distributed, multi-threaded system and around 10% of all unit tests and next 
> to all integration tests were affected by the exception you mentioned:
>
>> The exceptions are tests that use threads, and it's quite hard to do a good
>> job writing them.
>
> One can argue that several of the unit tests were not real unit tests because 
> they mocked some, but not all of the infrastructure they needed for running. 
> Anyway, they were running in Surefire (the integration tests in Failsafe), 
> and some lf them were flaky.
>
>> Just 'rerunning a few times' does not lead to useful
>> information in my view.
>
> Only the information that indeed they are flaky and a rough percentage of 
> cases in which they fail, which would be rather dependend on the test 
> environment though. Better than nothing information-wise. So far, so good.
>
> Having said all that, I still agree to Benson that a failing test should just 
> fail, period. I do not want developers to think, "it is only failing only in 
> 5% of all runs, so it is fine / low priority to fix". OTOH I understand 
> Qingzhou from a perspective of wishing to "test the test" or rather wishing 
> to harden a flaky test into a stable one, either by fixing the test or the 
> threading bug(s) it exposes. Benson's suggestion below looks promising 
> though, so thanks for that.
>
>> if you want random testing, there's a fine alternative at
>> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
>> maven plugin. It can already 'rerun your tests a few times' very easily.
>>
>> I wonder what the more involved surefire maintainers think, but as a PMC
>> member I'd be -1 on this proposal.
>>
>>> We think the right way to achieve this is to modify surefire plugin of
>>> maven. We want to add it as a part of the configuration of surefire, so
>>> users can decide whether they want to enable this feature, and how many
>>> times they want to rerun failing tests. We plan to open-source our
>>> contribution, and hopefully can merge our code into surefire master branch
>>> in the end. Therefore,  we are wondering do you have any
>>> comments/suggestions/opinions regarding this? We appreciate any input.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: Extend surefire to rerun failing tests

2014-05-28 Thread Alexander Kriegisch


-- 
Alexander Kriegisch
http://scrum-master.de

Am 28.05.2014 um 05:13 schrieb Benson Margulies :

> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo wrote:
> 
>> I am an intern at Google. The first step of my internship project is to add
>> the ability to Maven to automatically rerun failing tests a few times, to
>> see if they ever pass in any of the reruns. It is useful because in many
>> cases a test fails because it is flaky, not because there is a bug in the
>> new source code change.
> 
> I entirely disagree. 99.% of programs are completely deterministic. If
> you feed them the same inputs, you get the same output.

Well, in my previous project (I am an Agile Coach) my teams were working on a 
distributed, multi-threaded system and around 10% of all unit tests and next to 
all integration tests were affected by the exception you mentioned:

> The exceptions are tests that use threads, and it's quite hard to do a good
> job writing them.

One can argue that several of the unit tests were not real unit tests because 
they mocked some, but not all of the infrastructure they needed for running. 
Anyway, they were running in Surefire (the integration tests in Failsafe), and 
some lf them were flaky.

> Just 'rerunning a few times' does not lead to useful
> information in my view.

Only the information that indeed they are flaky and a rough percentage of cases 
in which they fail, which would be rather dependend on the test environment 
though. Better than nothing information-wise. So far, so good.

Having said all that, I still agree to Benson that a failing test should just 
fail, period. I do not want developers to think, "it is only failing only in 5% 
of all runs, so it is fine / low priority to fix". OTOH I understand Qingzhou 
from a perspective of wishing to "test the test" or rather wishing to harden a 
flaky test into a stable one, either by fixing the test or the threading bug(s) 
it exposes. Benson's suggestion below looks promising though, so thanks for 
that.

> if you want random testing, there's a fine alternative at
> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
> maven plugin. It can already 'rerun your tests a few times' very easily.
> 
> I wonder what the more involved surefire maintainers think, but as a PMC
> member I'd be -1 on this proposal.
> 
>> We think the right way to achieve this is to modify surefire plugin of
>> maven. We want to add it as a part of the configuration of surefire, so
>> users can decide whether they want to enable this feature, and how many
>> times they want to rerun failing tests. We plan to open-source our
>> contribution, and hopefully can merge our code into surefire master branch
>> in the end. Therefore,  we are wondering do you have any
>> comments/suggestions/opinions regarding this? We appreciate any input.

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org