| Magul added subscribers: Xqt, XXN, Dalba, valhallasw. Magul added a comment. |
The reason that (I believe) cause failing tests here (and in few other tasks) is, that these tests aren't unittests. The work on live Mediawiki instances and they assume that there will be no change during test.
As You can see in my previous comment it sometimes happend that change is made and it is failing.
I was trying to build here bullet proof solution. The solution mock BasePage.init method and decorate it logic with 2 lines that were responsible for getting last rev id for every page that was created during test and if it failed it would check if any page that was created during test changed. The problem here is that time required to run one such test was increased by at least one order of magnitude. Previously test take 3-4 seconds to pass and after implementing bullet-proof solution it was going for few minutes and haven't end.
So the question is what level of fragility is enough for us? I'm tending right now into:
- choping long tests that cuse problems to smaller ones
- on failure reruning test (that it will take 10 failures before test fails completely).
What do You think about it?
Cc: valhallasw, Dalba, XXN, Xqt, Magul, gerritbot, jayvdb, pywikibot-bugs-list, Aklapper, XZise, MayS, Mdupont, JJMC89, GPHemsley, Darkdadaah, Masti, Alchimista, Rxy, Krenair
_______________________________________________ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
