On Fri, Oct 30, 2015 at 11:11:52PM +0000, Guenter Milde wrote: > On 2015-10-28, Scott Kostyshak wrote: > > On Mon, Oct 26, 2015 at 03:33:22PM +0000, Guenter Milde wrote: > > ... > > >> However, if the exit status of an export test changes (from fail to > >> pass or vice versa), we should check whether this is due to a new bug, > >> a fix or just exposing previously hidden problems. > > > Agreed. In fact, sometimes it is an improvement when a test fails. When > > I check manually, it was passing before but showing garbled text in a > > PDF output. Now it might fail with a clear message of "language xyz not > > supported". It is always good to check manually why something fails and > > if it passes if the PDF output is good. > > And, instead of re-establishing this with every status change, we could > have "tags" for inverted tests, distinguishing: > > * failure because of known permanent incompatiblity (good failure) > e.g. lyx -e latex ... for a document using non-TeX fonts > > * failure because of ERT or preamble code (not LyX's fault) > > * failure because of upstream bugs > > * failure because of known LyX bugs > > * failure for unknown reason with non-standard export route > e.g. XeTeX and TeX-fonts
Yes this would be nice. Right now I just try to put that information as a comment for why we invert a test, but it would be nice to have that information more easily available in the test summary. Scott
