On Sunday, 8 June 2014, ogondza <[email protected]> wrote: > I agree with Ulli here. Upgrade tests is something that would be great to > have but turning ATH to what you describe does not make sense. > > 1) We _do_ want to know that version under test work on its own and not > only if you upgrade your config from some earlier version. >
I think you will find writing tests the way I envision gives that > > 2) If we start using config xml to configure/verify we are introducing > another interface that can and will change. As we have to keep all our page > objects there will be more breakage due to changes in SUT than before > (changes in persistence does not affect ATH now). > But the breakage will happen at different times, thus you retain more confidence that the tests you have written are not giving a false positive. The xml schema changes, you fix the test and it passes again... That test is driving via page objects and verifying via xml, so the risk of a false positive from changing the "xml objects" is low. The UI changes, you fix the test and it passes again... That test is driving via xml and verifying via UI, since the xml is the same you know that the risk of introducing a false positive is lower... Another test is driving via the UI and verifying via xml... When that test returns to passing as a side effect of changing the page object to fix your first test, because the xml object remains unchanged we know the risk of false positive is also low. When you drive and verify by the same axis you need to recheck each test by hand to ensure a false positive has not crept in... Or take a risk assessment and cross your fingers. > > 3) What you suggest allows test creator to create test fixture using > _some_ core/plugin version. Whole test suite will then inconsistently test > if upgrade works from different version for every test. I expect more > serious approach from upgrade tests.* Not to mention that such tests will > be close to useless a year from now as it will be performing upgrades most > users already performed. > No actually you create xml objects that drive the config file creation. So we can update these as the schema changes... Much like with the page objects for the UI... For upgrade testing you can request the page objects to output the older schema version > > > 4) Having one test failure for one change seem nice in theory but after a > year and half of fixing selenium tests to reflect UI changes I am not > really motivated to work against that goal. > This is about significantly reducing the risk of false positives. That requires picking different axes to drive vs verify. Once you adopt that philosophy the effect is that the nature of the test failures vs test errors naturally fall out. > > > * What I would like to make sure is users can upgrade from LTS to LTS-rc > and from master to master-rc preserving the same functionality. We should > use existing POs to setup fixture, then upgrade Jenkins/plugins and execute > WHEN and THEN using newer version. This comes for free as we are > maintaining page objects already > The point that you seem to be missing is *every* time you change a page object you risk turning each test that uses the page object into a false positive. This is not a theoretical risk. It is a very real risk I have personal experience of. When you "just update and maintain the page objects" you need to recheck *every* test that uses the page object to ensure it is still providing a valid test. Short of using something like jester (of old) or some other mutation engine to actually break Jenkins and verify that the rest then fails this is not something easily automated. That is my experience. If people want to ignore it, well fine, I'll shut up now. Do not expect me to contribute any UI driven tests that verify via the UI... If that means my tests are not part of the a-t-h... So be it... If the community does not understand my point, let me know and I will try to explain it better... The TL;DR version of my point... Just in case you missed it... Don't drive the test and verify the test result by the same axis, as the risk of a false positive is significantly higher especially in a changing code base. > , no one need to update XML fixtures regularly to cover upgrade most > people will undergo soon and adding new use case like upgrading to last LTS > release line from the previous one is just a matter of test > parameterization. > > -- > oliver > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected]');> > . > For more options, visit https://groups.google.com/d/optout. > -- Sent from my phone -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
