On Fri, Jan 20, 2012 at 1:43 AM, Clint Byrum <cl...@ubuntu.com> wrote:
> Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012: > > Hi! > > > > > - According to this website there are still 94 test failures in 5.4 . > > > Can you confirm all of them are minor problems? > > > http://gcov.php.net/viewer.php?version=PHP_5_4 > > > > Most of them appear so, I'll go through them again to be sure and > > encourage others to do so too and raise red flags if somebody sees > > something bad there. > > Unfortunately, some tests are environment-dependent or otherwise have > > subtle dependencies or structure that make them work on one system and > > fail on another not because of the bug in PHP but because of the test > > itself. So, I have 0 fails on my Linux build but 6 fails on my Mac > > build. Other times some systems may not support some capability, use old > > version of the library, etc. and the test may not account for that. > > These tests should be skipped or marked as XFAIL on platforms they are > known to fail on. Better to have no test than one that cannot be relied > upon. All supported platforms should pass with 0 fails. These intentional > skips should have open bugs that are documented in the test code so that > a developer can find out why this test was disabled when trying to make > a change covered by the test. > > > > > I do not think it is practical to postpone release until we solve all of > > such problems, since this being volunteer-driven open-source project > > this means not having any release schedule at all. I prefer having the > > schedule even if that means we'd have to release with some known > > deficiencies. > > > > Its pretty bad actually. For all of PHP's success, this is something that > continues to baffle me, and many others I have talked to who are charged > with measuring quality and with patching systems in a timely manner. How > better to document unreliable tests than to skip them with something like > "SKIPPED - known to fail on Mac". > > Its precisely this unreliability that forced me to take a conservative > approach for Ubuntu 12.04 and recommend to the community that we ship > 5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but > even if all the tests pass on the machine we run the test suite on, > how can we be sure they won't fail in another time zone, or in some > other strange configuration? > Given that 12.04 beta2 will be out on Thursday, and the unit tests where fixed before shipping 5.4 (I naively assume), is this in or out? ref: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-php54 With beta freeze on march 22 I guess the "mistake" of bundeling 5.3 instead of 5.4 is already made, and we will have to live with that for the next 2 years for prod, and the next 7 months for dev. > > > > - There was this problem with 5.3.7 and the crypt() bug. Has there been > > > some improvement in the process of handling test failures? For example > > > mark expected failures as expected failures, and fix the tests or the > > > code? Or are the failing tests "stable" since month and all of them are > > > minor problems? > > > > Yes, there was work done on these. Most of those were fixed, but few > > still remain, especially across various environments (i.e. test may be > > fine on some but not others). I of course am all for fixing that further > > and welcome any help on that. > > > > > - There have been 319 unique failed tests in RC5, reported by user > > > tests. Is someone looking into them and trying to classify and/or fix > them? > > > http://qa.php.net/reports/ > > > > Non-reproducible failures usually mean the problem is with the test > > itself, or with the difference of expectations in the test and local > > environment, not with PHP. It may still be PHP problem, of course, so > > the person running the test should check it out and submit a bug if > > appropriate and if it's bad enough, also send a message to this list. > > > > > All in all the number of test failures still feels very high, I would > be > > > interested in your opinion. Is this "normal" for big projects like > this? > > > > I do think it should be reduced, however if the choice is between > > waiting forever and have release with some bugs, I think it is practical > > to choose the latter. Of course, if we discover a major problem that > > makes PHP unusable or seriously impacts many PHP users, it will be dealt > > with immediately, and had been so in the past, but otherwise we have to > > work within the realities of a big project with limited resources and > > realize that while we strive for 0 bugs in every release, it may never > > be possible. > > All software will have bugs. The test suite, however, should reflect > the bits of code that you know work reliably... not the bits of code > you know work most of the time. > > The fact that its all being running regularly is a fantastic improvement. > I'd like to see a commitment to getting 100% pass/xfail/skip for every > release/tested environment in future releases though. > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
