Re: [Wikitech-l] known to fail switch added to parserTests
Hoi, Suppose that someone fixes a test that has been always failing... one of those known to fail. It makes no difference right ?? Giving them the status of pass is imho dead wrong because they should not fail in the first place.. now a status of KNOWN TO FAIL makes sense. Thanks, GeardM 2009/7/20 dan nessett dness...@yahoo.com I have modified parserTests to take a known to fail switch so those tests that have always failed now pass. It was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I added an option for each test called flipresult. When this option is specified, the test succeeds when it fails and vice versa. I have tested the modified parserTest on 1.16a running over a 1.14 schema database. However, I have run into a problem attempting to install the latest trunk revision so I can test against it. Specifically, I added a database called wikitestdb so I would have a production and test wiki. However, when I checked out the latest trunk revision, ran the install script and update.php, and then accessed http://wiki path/index.php the installation gets into a infinite redirect loop. When I attempted to debug this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is, Main_Page is rendered and displayed. The only difference between the two URLs are the first uses http://wiki path/index.php (which redirects to http://wiki path/index.php/Main_Page), while the debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDEBUG_SESSION_START=netbeans-xdebug . I need some help figuring out what is happening. I imagine using this list for that purpose would be inappropriate. So, if someone would volunteer to help me (email me at the from address in this email), I can get the parserTest changes tested against the newest revision. I can then open a bug (or use an already open bug) and attach the patch and edited parserTests.txt file to it. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 7:19 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: Suppose that someone fixes a test that has been always failing... one of those known to fail. It makes no difference right ?? Difference in what sense? It means we have one less failing test reported, presumably. Giving them the status of pass is imho dead wrong because they should not fail in the first place.. now a status of KNOWN TO FAIL makes sense. The known-failing tests have never passed. They're a wishlist. None of them are likely to be fixed in the foreseeable future. I'd be fine with just removing them, but Brion has been against it in the past. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
Hoi, There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of known to fail makes sense, just changing them to pass makes it necessary to add a citation needed because it is just not true. Thanks, GerardM 2009/7/21 Aryeh Gregor simetrical+wikil...@gmail.comsimetrical%2bwikil...@gmail.com On Tue, Jul 21, 2009 at 7:19 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: Suppose that someone fixes a test that has been always failing... one of those known to fail. It makes no difference right ?? Difference in what sense? It means we have one less failing test reported, presumably. Giving them the status of pass is imho dead wrong because they should not fail in the first place.. now a status of KNOWN TO FAIL makes sense. The known-failing tests have never passed. They're a wishlist. None of them are likely to be fixed in the foreseeable future. I'd be fine with just removing them, but Brion has been against it in the past. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of known to fail makes sense, just changing them to pass makes it necessary to add a citation needed because it is just not true. Nobody's changing them to pass. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 7:43 AM, Kwan Ting Chank...@ktchan.info wrote: Aryeh Gregor wrote: On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of known to fail makes sense, just changing them to pass makes it necessary to add a citation needed because it is just not true. Nobody's changing them to pass. I can understand where Gerard got the impression: I have modified parserTests to take a known to fail switch so those tests that have always failed now pass. - dan nessett 2009-07-20 16:09 KTC -- Experience is a good school but the fees are high. - Heinrich Heine ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l That being said, a patch against 1.14.x is of no use to anyone. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
Hoi, That is exactly the problem. You report that they pass and in reality they still fail. Someone should change them to pass ie fix the software. Thanks, GerardM 2009/7/21 Aryeh Gregor simetrical+wikil...@gmail.comsimetrical%2bwikil...@gmail.com On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of known to fail makes sense, just changing them to pass makes it necessary to add a citation needed because it is just not true. Nobody's changing them to pass. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 3:21 PM, Gerard Meijssengerard.meijs...@gmail.com wrote: Hoi, That is exactly the problem. You report that they pass and in reality they still fail. Someone should change them to pass ie fix the software. I think the appropriate English reply in this context would be No s**t, Sherlock... ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
The change I made was to add a flipresult option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open. I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either. Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow? --- On Tue, 7/21/09, Kwan Ting Chan k...@ktchan.info wrote: From: Kwan Ting Chan k...@ktchan.info Subject: Re: [Wikitech-l] known to fail switch added to parserTests To: Wikimedia developers wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 4:43 AM Aryeh Gregor wrote: On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijs...@gmail.com wrote: There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of known to fail makes sense, just changing them to pass makes it necessary to add a citation needed because it is just not true. Nobody's changing them to pass. I can understand where Gerard got the impression: I have modified parserTests to take a known to fail switch so those tests that have always failed now pass. - dan nessett 2009-07-20 16:09 KTC -- Experience is a good school but the fees are high. - Heinrich Heine -Inline Attachment Follows- ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Wed, Jul 22, 2009 at 12:45 AM, dan nessettdness...@yahoo.com wrote: I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either. Having a third possible result would be more informative. You could report two scores, one including known-to-fail tests and one excluding them. Reporting both or just one of these scores could be configurable by a command line option. -- Stephen Bain stephen.b...@gmail.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 10:45 AM, dan nessettdness...@yahoo.com wrote: The change I made was to add a flipresult option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open. This isn't a good idea, no. The important thing is if someone runs parserTests.php, they should be able to easily tell whether there are any regressions in their working copy. But if we're going to keep the known-to-fail tests at all, it doesn't make a lot of sense to report them as passing when they're actually failing . . . if we do that we may as well just drop them. I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either. It should be made clear that some tests are failing, but that they are not regressions. Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow? I don't know offhand. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 10:51 AM, Aryeh Gregor simetrical+wikil...@gmail.com simetrical%2bwikil...@gmail.com wrote: But if we're going to keep the known-to-fail tests at all, it doesn't make a lot of sense to report them as passing when they're actually failing . . . if we do that we may as well just drop them. The tests have never passed - they should be commented out for usability reasons. And ideally there would be a post-commit hook that runs the parser tests and e-mails the committer letting them know they have just broken the software! ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 12:57 PM, Brianbrian.min...@colorado.edu wrote: The tests have never passed - they should be commented out for usability reasons. Well, I have no objections, but apparently that's not acceptable. An expected fail flag would be about as usable. And ideally there would be a post-commit hook that runs the parser tests and e-mails the committer letting them know they have just broken the software! Yes, that would be nice. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message. --- On Tue, 7/21/09, Brian brian.min...@colorado.edu wrote: From: Brian brian.min...@colorado.edu Subject: Re: [Wikitech-l] known to fail switch added to parserTests To: Wikimedia developers wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 9:57 AM On Tue, Jul 21, 2009 at 10:51 AM, Aryeh Gregor simetrical+wikil...@gmail.com simetrical%2bwikil...@gmail.com wrote: ... And ideally there would be a post-commit hook that runs the parser tests and e-mails the committer letting them know they have just broken the software! ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
On Tue, Jul 21, 2009 at 12:05 PM, dan nessett dness...@yahoo.com wrote: Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message. Right, well, a pre-commit hook that rejects all commits which break the software. Or a memory of what commits broke which tests and a conditional. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] known to fail switch added to parserTests
Better ideas. Another possibility is every 24hrs to run parser tests (and any other regression tests that might exist) against all revisions committed into trunk since the last run. Post the results and keep track of the number of bugs each committer has introduced into the code base for the past running 6 month period. Post the names of committers and the number of bugs they have introduced on a hall of shame page ordering the list by number of bugs. Sometimes social pressure can be a very effective behavior modifier. --- On Tue, 7/21/09, Brian brian.min...@colorado.edu wrote: From: Brian brian.min...@colorado.edu Subject: Re: [Wikitech-l] known to fail switch added to parserTests To: Wikimedia developers wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 11:10 AM Right, well, a pre-commit hook that rejects all commits which break the software. Or a memory of what commits broke which tests and a conditional. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
