Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012: > Hi! > > > - According to this website there are still 94 test failures in 5.4 . > > Can you confirm all of them are minor problems? > > http://gcov.php.net/viewer.php?version=PHP_5_4 > > Most of them appear so, I'll go through them again to be sure and > encourage others to do so too and raise red flags if somebody sees > something bad there. > Unfortunately, some tests are environment-dependent or otherwise have > subtle dependencies or structure that make them work on one system and > fail on another not because of the bug in PHP but because of the test > itself. So, I have 0 fails on my Linux build but 6 fails on my Mac > build. Other times some systems may not support some capability, use old > version of the library, etc. and the test may not account for that.
These tests should be skipped or marked as XFAIL on platforms they are known to fail on. Better to have no test than one that cannot be relied upon. All supported platforms should pass with 0 fails. These intentional skips should have open bugs that are documented in the test code so that a developer can find out why this test was disabled when trying to make a change covered by the test. > > I do not think it is practical to postpone release until we solve all of > such problems, since this being volunteer-driven open-source project > this means not having any release schedule at all. I prefer having the > schedule even if that means we'd have to release with some known > deficiencies. > Its pretty bad actually. For all of PHP's success, this is something that continues to baffle me, and many others I have talked to who are charged with measuring quality and with patching systems in a timely manner. How better to document unreliable tests than to skip them with something like "SKIPPED - known to fail on Mac". Its precisely this unreliability that forced me to take a conservative approach for Ubuntu 12.04 and recommend to the community that we ship 5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but even if all the tests pass on the machine we run the test suite on, how can we be sure they won't fail in another time zone, or in some other strange configuration? > > - There was this problem with 5.3.7 and the crypt() bug. Has there been > > some improvement in the process of handling test failures? For example > > mark expected failures as expected failures, and fix the tests or the > > code? Or are the failing tests "stable" since month and all of them are > > minor problems? > > Yes, there was work done on these. Most of those were fixed, but few > still remain, especially across various environments (i.e. test may be > fine on some but not others). I of course am all for fixing that further > and welcome any help on that. > > > - There have been 319 unique failed tests in RC5, reported by user > > tests. Is someone looking into them and trying to classify and/or fix them? > > http://qa.php.net/reports/ > > Non-reproducible failures usually mean the problem is with the test > itself, or with the difference of expectations in the test and local > environment, not with PHP. It may still be PHP problem, of course, so > the person running the test should check it out and submit a bug if > appropriate and if it's bad enough, also send a message to this list. > > > All in all the number of test failures still feels very high, I would be > > interested in your opinion. Is this "normal" for big projects like this? > > I do think it should be reduced, however if the choice is between > waiting forever and have release with some bugs, I think it is practical > to choose the latter. Of course, if we discover a major problem that > makes PHP unusable or seriously impacts many PHP users, it will be dealt > with immediately, and had been so in the past, but otherwise we have to > work within the realities of a big project with limited resources and > realize that while we strive for 0 bugs in every release, it may never > be possible. All software will have bugs. The test suite, however, should reflect the bits of code that you know work reliably... not the bits of code you know work most of the time. The fact that its all being running regularly is a fantastic improvement. I'd like to see a commitment to getting 100% pass/xfail/skip for every release/tested environment in future releases though. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
