Well, I think it would help if especially a team that develops a CI server would use the TDD approach to introduce new changes (eat your own dogfood!). So unrelated to the discussion if system tests (or acceptance tests) are valuable I would never try to commit new changes (especially large changes as you are describing) before all tests are GREEN. This is a basic principle that we should use as Jenkins dev team. So if a tests breaks due to a change, we should fix the bug or test (in the LDAP case the test needed to get an update to the new LDAP UI configuration, this is already fixed now - thanks to the work of Michael…)
The use case you are describing (i.e. introduce additional integration tests that work on injected xml files) is a totally different one and makes sense too. I don’t see that we need to decide between two options, both are valid. But since we are mostly part time developers we should focus only on things we can actually handle in our valuable spare time. So I would suggest to improve the existing UI tests first until we think they are in a good shape before we continue with the next area of integration tests. I.e., one step after the other… I have also seen projects dumping their UI tests due to the amount of work required to keep them up to date with the development (mostly due to lacking framework functionality). But I also worked in projects that got these test evolving with the underlying systems without any problems. This is a matter of design of the UI and the test cases and I think we will manage this in Jenkins and keep our test suite up to date without much work). Am 04.06.2014 um 23:28 schrieb Stephen Connolly <[email protected]>: > > > On Wednesday, 4 June 2014, oliver gondža <[email protected]> wrote: > Stephen, > > You suggest strictly separating UI tests and behavior tests, but using > backdoor manipulation/verification via xml configuration is no longer > end-to-end testing. Not to mention that plugin config is not exposed to user > as XML. > > But the xml file is what the user has when they upgrade. > > I am suggesting breaking the tests into two. Splits the bits that generate > the config from the hits that test the config works. You still have end to > end tested, but in two steps and you now have tests that don't need to change > with the UI. > > I have seen the pattern of UI driven tests going end to end fail over and > over as the system under test evolves. > > Further it becomes a pain to re validate all the many many test cases as the > page objects are updated to ensure that you have not introduced false > positives. This is what eventually leads to EITHER all the tests being thrown > out and rewritten from scratch OR changes in the UI being rejected because > they break too many tests... > > Not to mind that the number one question people have is: > > "If I upgrade my Jenkins will all my existing jobs still work the same" > > Contrast with the current pure UI driven tests that answer a different > question: > > "If I download the new version of Jenkins and install it on a new server, can > I configure new jobs that behave the same as the jobs I had on my old Jenkins" > > I argue that this second question, while important, is nowhere even within an > order of magnitude close to the importance of the first question. > > Plus, if we move test fixtures to xml and the plugin persistence evolves (as > in this case), we are testing that plugin is able to load obsolete config and > not that it can be configured to do the same using its new version. > > Yes which is why you need to add new tests for the new configuration > schema... Plus if the configuration changes you will have the UI test fail as > they will fail to generate the xml configured as before, so you get an alert > of this case. > > > Also note that with all the test we currently have, only a small part of > Jenkins UI is covered. The goal of covering UI in small portion of tests and > then focusing on the behavior does not sound achievable to me. In effect we > will have less UI coverage. > > Well the point is the current tests are only testing the UI in respect of the > WHEN-THEN parts of the tests. You are fooling yourself if you think the > exercising of the UI in the GIVEN part of the test is testing the UI. > > I am recommending that the GIVEN part of the tests not drive by UI. Thus we > will not have a lower actual testing of the UI... We will have the same > amount of UI coverage and we will have removed the false sense of security > people currently are giving themselves with the GIVEN exercising of the UI > > > -- > oliver > > > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: Message signed with OpenPGP using GPGMail
