A lot of comments and valuable feedback has popped up by the end of the
week. Let me react here instead of on individual messages.
- No argument on slow test run, high flakiness, fragility. Some of that
is caused by not enough attention as ATH stands a bit aside the delivery
pipeline. Bringing it to PR builds of core or plugins would point more
people to it (hopefully making it better) but it would be annoying at
first until we make it more mature/stable. The question is if we are
willing to bite the bullet here.
- With regards to the *kinds* of tests that should be part of ATH, the
thing is this was never defined so different kinds got in. I tend to
agree that what can be written against JTH, should. Perhaps it would be
beneficial to write that down and guard no more such tests are added to
prevent further bloating. The existing ones can either be deleted,
migrated or tolerated based on how much effort does it take to maintain
them.
- Ad smoke tests: This seems like a good compromise (starting point?)
for PR integration for several reasons. No question the actual test set
should be revisited. All (sufficiently distinct) core scenarios +
recommended plugin set scenarios + some popular scenarios of
non-recommended plugins?
- Speaking of deleting flaky tests - I am willing to give this a try as
the idea of stable build sounds wonderful and almost unheard of in ATH
land. As a maintainer of downstream plugin update center certified by
ATH internally, I am less excited. Again, I would like to see some
consensus on the process. Sometimes it might be enough to check JTH
covers same scenario or it can be migrated easily. What if it does not
and the scenario appears fundamental? How to tell?
Please let me know if there are any additional thoughts or I have not
commented on something. Later this week, I intend to summarize this in a
doc or two in ATH repo to base our future decisions on (w.r.t. removing
tests, not accepting them, etc.) and update the thread with PR link.
--
oliver
--
You received this message because you are subscribed to the Google Groups "Jenkins
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/6379b1ca-a85d-9041-1695-e9cd02aa042b%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
