I would prefer B On Tue, Jan 31, 2017 at 1:34 PM, Igor Fedorenko <i...@ifedorenko.com> wrote:
> > B. Fix the test, but exclude the broken versions of Maven from the range > with a comment explaining why > > I sometimes rerun integration tests against released versions of Maven > to validate the tests are still working and I know other developers who > do that too. Having failures would just mean tests are broken and can be > ignored IMHO. > > -- > Regards, > Igor > > On Tue, Jan 31, 2017, at 06:42 AM, Stephen Connolly wrote: > > We have kind of established a consensus on how to handle the case where > > we > > want to change the specification of how Maven works going forward. > > Specifically, if we decide that the old behaviour of Maven is no longer > > going to be the new behaviour of Maven our procedure in the integration > > tests is as follows: > > > > 1. Mark the existing tests that are affected as range limited where the > > upper bound is the below the version of Maven that the change in > > behaviour > > will land in > > 2. Create tests of the new behaviour (probably copied from the original > > tests but with the assertions modified and using a range limited where > > the > > lower bound is the version of Maven that the change in behaviour will > > land > > in. > > > > An example of such a change is > > https://github.com/apache/maven-integration-testing/commit/ > c4365abe20b58b2cbc174de812e43c7741dc10e1 > > > > We now have a more complex case to try and decide how to handle, the > > current attempt to resolve is this diff: > > https://github.com/apache/maven-integration-testing/ > compare/master...MNG-2199 > > > > However I am somewhat uncomfortable with how that proposed fix to the > > integration tests works. > > > > So firstly, Christian has identified that the original tests added were > > not > > correctly detecting the failure. > > > > We have a situation therefore where the integration tests have been > > giving > > false positive results against Maven 3.2.2+ > > > > Therefore, my view is that we should *fix the broken tests* because a > > false > > positive or a false negative is a bug in the tests. > > > > This would mean that the tests would no longer pass when run against > > 3.2.2-3.3.9, instead they would report the bugs in those versions that we > > shipped due to the bugs in the integration tests. > > > > If we had a need to release - say security fixes - for those lines, we > > would then have to do one of: > > * ACK the continued failing tests; > > * Run with the integration tests forked from the point in time where the > > previous release on the line was cut; OR > > * Back-port the fixes to those lines > > > > (assuming we are supporting those lines for security fixes) > > > > I am fine with any of those three options as those are known issues that > > we > > should really have JIRAs for and be documenting in the release notes, and > > any of those three options would be forcing us to acknowledge the bugs. > > > > An alternative is to say "those bugs were part of the specification of > > Maven and we have changed the specification of Maven again" which is the > > approach that the current MNG-2199 branch takes. > > > > I am not happy with that approach as it is an implicit approval of that > > type of usage for the broken versions of Maven. Users could legitimately > > start filing feature requests to "restore" the previous behaviour because > > "it was part of the specification"... fine we can probably bat those > > requests away, but is it helping us with code archeology? > > > > So, what do we want to do with the case of a test being identified as > > having either a false positive or a false negative against an already > > released version of Maven? > > > > A. Fix the test and then the test will fail against already released > > versions of Maven > > B. Fix the test, but exclude the broken versions of Maven from the range > > with a comment explaining why > > C. Clone the test, leaving the broken test for the old versions of Maven > > and the new test for new versions of Maven > > D. Something else > > > > I personally favour A or B (with a slight leaning towards A) and I really > > do not like C for the case of the false-positive / false-negative tests > > > > If an obvious consensus does not emerge I may have to call a vote, you > > have > > been warned! > > > > -Stephen > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > > -- ----- Arnaud Héritier http://aheritier.net Mail/GTalk: aheritier AT gmail DOT com Twitter/Skype : aheritier
