It appears there is no clear consensus in the developer community how to
resolve the problem of the parserTests known-to-fail tests. Consequently, the
most useful thing I can do is provide the patches I have developed and explain
how they can be used to implement one of a number of options. These are neither
the only options nor is one of them necessarily the best. They simply
correspond to suggestions made on the wikitech-1 email list (the second option
is additional). There is, of course, the possibility of developing an option
not specified here and implementing it.
[Mark Clements added another option this morning - 7/27/09 - that requires
additional development. If that is what people want, I (or others) can do the
necessary implementation. However, I thought it best to get out what I have
done so far in case it meets consensus needs.]
I have attached the patches to bug 19957 (I couldn't find an appropriate
existing bug for this issue, so I created one. Also, I attached a tar file with
7 patches. When I looked at the bug after submitting it, it wasn't clear the
tar file made it. Any advice?). These patches can be used to implement the
options specified below. Details of the patches are found in text I have added
to the bug report.
+ Install none of these patches and do nothing else:
+ Pros: parserTests behavior remains as currently defined.
+ Cons: parserTests behavior remains as currently defined.
+ Install none of these patches and define successful output as that which the
Parser currently returns (this would require changing parserTests.txt using a
unprovided patch):
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It is generally bad practice to define success without understanding
the logic behind it.
+ Install none of these patches, file bugs against the 14 failing tests and
then fix the bugs:
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It has been suggested that the parserTests logic is sufficiently
complex that it may be difficult to modify it to pass the known to fail tests.
+ File bugs against the 14 failing tests and patch parserTests.txt to comment
out the known to fail tests:
+ Pros: Testing the Parser using parserTests does not return confusing test
results. Developers can concentrate on those tests that fail due to the
addition of new functionality or intefering bug fixes.
+ Cons: Since the known to fail tests no longer report their presence, it may
be easy for the community to forget they are still a problem.
+ Patch parserTests.txt so the known to fail tests are disabled. Optionally
file bugs against these tests:
+ Pros: Testing the Parser using parserTests does not return confusing test
results. Developers can concentrate on those tests that fail due to the
addition of new functionality or intefering bug fixes.
+ Cons: There are other disabled tests in parserTests.txt. These could become
confused with the known to fail tests.
+ Patch parserTests|.inc|.php to provide a --run-disabled option:
+ Pros: This would allow disabled known to fail tests to run without editing
parserTests.txt
+ Cons: Specifciation of this option runs all disabled tests, not just those
known to fail. Test results using this option might confuse developers.
+ Patch parserTests|.inc|.txt to provide a knowntofail option:
+ Pros: parserTests output and summary statistics indicate which tests
succeeded, failed and returned known-to-fail status.
+ Cons: Creates a new test status for what should be a temporary problem.
This is a questionable software architecture decision.
+ Patch parserTests.php to support a ktf-in-fail option:
+ Pros: those of the previous option. Adds the ability to accumulate ktf
failure status in failure statistics.
+ Cons: those of the previous option. Raises the question of what is the
proper statistics strategy when the known-to-fail option is missing (on runs
with this option missing, known-to-fail tests accumulate against success
statistics).
The provided patches do not address whether the 'testrun' and 'testitem' tables
should be modifed to note the specification of statistics changing application
options (i.e., either --ktf-in-fail or --run-disabled) or the use of the
knowntofail option for a particular test. These tables are not modified by the
patches.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
