Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
C. Scott Ananian wrote: On Thu, 17 Mar 2005, Jakob Eriksson wrote: Ouch. Of course, should have thought of that. Maybe we need a patch penguin? CVS actually handles 'vendor branches' fairly nicely. It should be possible for Jakob (or whoever else decides to be the 'test penguin') to maintain his own 'testing' CVS tree, with rapid development, etc, and periodically resync his own tree to canonical Wine CVS (using the vendor branch functionality) and then create large-ish 'tests-only' chunks from that to throw at Alexandre "once in a while". There really shouldn't be much reason for Alexandre to reject patches that touch tests only; after all, if the tests pass on windows, they should pass on wine, no matter how evil they look. (Well, within reason.) That's MHO, at least. If I understand correctly, the primary reason for the 'testing' CVS is just to manage distribution of proposed tests to a server farm of test-runners; which is slightly different from the purpose of the mainline CVS tree. [Also -- a decoupled 'testing' CVS like I describe above can be implemented by the motivated folks w/o Alexandre's involvement at all, which permits judgements to be postponed until we've got some evidence of usefulness.] Yes. And I think I can implement most of even the more elaborate schemes without initially disturbing Alexandre or anyone else. As you say, until we get more evidence of usefulness. In this vein -- where *is* the current testing infrastructre located? I'm pretty new to Wine, and I couldn't find any links from winehq. [These should probably be added, or made more visible if they do exist, especially if the goal is to encourage test submission with patch submission.] http://test.winehq.org/data/ regards, Jakob
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Paul Millar wrote: On Thursday 17 March 2005 11:54, Jakob Eriksson wrote: Maybe one could script something up so that a commit to tests CVS would not be *possible* without a confirmed test pass on Windows 95, NT, 2000 and XP. That'd be the best thing since sliced bread. CVS supports doing some server-side validation tests before accepting a commit, so it could be done; but I don't think this would be "nice". CVS is a mechanism for sharing code: its not really a testing framework. I didn't mean exactly on the CVS level. When Alexandre commits any patch, he first checks that the code still passes regression tests. I want something similar for the test patches. Maybe like this: the testing dispatcher signs a working patch* with GPG. (Or no GPG. Just set a flag somewhere. Details are not important.) Alexandre will see this flag when saving a patch from [EMAIL PROTECTED] and know that the patch is OK as far as the test grid is concerned. It could be as non-intrusive as this: the test dispatcher monitors the [EMAIL PROTECTED] for patches. As soon as it sees a patch it recognises and knows it has tested, it sends a mail to wine-patches akin to: The patch with CRC32 so-and-so posted by him or her, named so-and-so is hereby verified by me, the Wine Regression Grid Tester. ** or The patch with CRC32 so-and-so posted by him or her, named so-and-so failed under the following versions of Windows; bla bla blah, with the following error message: blah blah bla some more. Truthfully yours, the Wine Regression Grid Tester. ** Then it's up to Alexandre if he wants to commit a test which the grid tester has rejected, or for which there is no confirmation. If you don't like the idea of a program spamming wine-patches, it could be separate list, or a webpage with a copy of wine-patches, with different messages colour-codes updated as they get tested by the grid tester. * A working patch is a patch that has been tested and found working on Win 95, 98, ME, NT4, 2000 and XP. ** Could we call it WineGrind? :-) For the testing framework, I'd say what we have just now is fine. It lives outside (and on top of) CVS. Having broken tests is OK, provided they're fixed within a suitable time-scale [*]. Actually, I think having broken tests is not OK. It not only goes against my zealotry for Extreme Programming, it's also very annoying when I have _no_clue_ how to fix a broken test and the author is missing or don't want to touch the code with a ten foot stick. (just my 2c-worth again) Cheers, Paul [*] -- Of course, what is the "suitable time-scale"? who's willing to make sure things get fixed? Exactly, I feel rage everytime I see those red and yellow boxes at http://test.winehq.org/data/ ;-) regards, Jakob
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
On Thu, 17 Mar 2005, Jakob Eriksson wrote: Ouch. Of course, should have thought of that. Maybe we need a patch penguin? CVS actually handles 'vendor branches' fairly nicely. It should be possible for Jakob (or whoever else decides to be the 'test penguin') to maintain his own 'testing' CVS tree, with rapid development, etc, and periodically resync his own tree to canonical Wine CVS (using the vendor branch functionality) and then create large-ish 'tests-only' chunks from that to throw at Alexandre "once in a while". There really shouldn't be much reason for Alexandre to reject patches that touch tests only; after all, if the tests pass on windows, they should pass on wine, no matter how evil they look. (Well, within reason.) That's MHO, at least. If I understand correctly, the primary reason for the 'testing' CVS is just to manage distribution of proposed tests to a server farm of test-runners; which is slightly different from the purpose of the mainline CVS tree. [Also -- a decoupled 'testing' CVS like I describe above can be implemented by the motivated folks w/o Alexandre's involvement at all, which permits judgements to be postponed until we've got some evidence of usefulness.] In this vein -- where *is* the current testing infrastructre located? I'm pretty new to Wine, and I couldn't find any links from winehq. [These should probably be added, or made more visible if they do exist, especially if the goal is to encourage test submission with patch submission.] --scott LIONIZER class struggle SCRANGER ESGAIN overthrow GRALLSPICE UKUSA tonight EZLN Sudan NSA KMFLUSH PBPRIME mail drop Leitrim IDEA AELADLE ( http://cscott.net/ )
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Paul Millar wrote: On Thursday 17 March 2005 11:51, Jakob Eriksson wrote: Paul Millar wrote: [writing tests]'s a rather weary job, which no one enjoys doing :^/ I do! Maybe I have a condition, but I really love doing it! :-) Long may that continue! Thanks, I hope so too! :-) Only my day job and things like planning a marriage stops me from digging deep into Wine testing. [..] I'd tread careful here: this isn't a panacea. Yes, its fairly easy to *say* OK, we just merge in the new tests, but would merging the snapshots really be a trivial job? If it is non-trivial, then Alexandre would front at least some of it and I suspect making more work for Alexandre is the last thing anyone (including Alexandre :^) would want. Ouch. Of course, should have thought of that. Maybe we need a patch penguin? regards, Jakob
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
On Thursday 17 March 2005 11:51, Jakob Eriksson wrote: > Paul Millar wrote: > >[writing tests]'s a rather weary job, which no one enjoys doing > > :^/ > > I do! Maybe I have a condition, but I really love doing it! :-) Long may that continue! [...] > > [separate CVS tree] > This makes sense. Winetest could actually be a separate project. > I'm not saying it should be, mind you, just that conformance > testing is interesting not only for Wine, but for Windows > developers in general. > > So a separate CVS tree makes a great deal of sense, IMHO. > Wine could import snapshots of the tests for its' conformance > testing. This could speed up test development considerably, I > imagine. I'd tread careful here: this isn't a panacea. Yes, its fairly easy to *say* OK, we just merge in the new tests, but would merging the snapshots really be a trivial job? If it is non-trivial, then Alexandre would front at least some of it and I suspect making more work for Alexandre is the last thing anyone (including Alexandre :^) would want. On top of that, there would be many other issues, e.g. How the diffs between the two trees were presented? How the changes in wine would be merged back into the winetest tree? What happens to patches that Alexandre rejects (coz they're just wrong)? Ultimately, its Alexandre's call (and I think I can guess his answer). [...] > Are you coming to wineconf? All things being equal, yes. Cheers, Paul. pgpJPmyTe9YX6.pgp Description: PGP signature
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Ivan Leo Puoti wrote: Jakob Eriksson wrote: Maybe one could script something up so that a commit to tests CVS would not be *possible* without a confirmed test pass on Windows 95, NT, 2000 and XP. That would be very hard to do and mostly pointless for drivers. Sometimes hard is worthwile. At least to me. regards, Jakob
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
On Thursday 17 March 2005 11:54, Jakob Eriksson wrote: > Maybe one could script something up so that a commit to > tests CVS would not be *possible* without a confirmed test > pass on Windows 95, NT, 2000 and XP. > That'd be the best thing since sliced bread. CVS supports doing some server-side validation tests before accepting a commit, so it could be done; but I don't think this would be "nice". CVS is a mechanism for sharing code: its not really a testing framework. There would be potential issues with CVS locking, timeouts (e.g. what happens if the XP machine dies?), but the real issue is it would just gets in the way of developers by making the CVS server fragile. For the testing framework, I'd say what we have just now is fine. It lives outside (and on top of) CVS. Having broken tests is OK, provided they're fixed within a suitable time-scale [*]. (just my 2c-worth again) Cheers, Paul [*] -- Of course, what is the "suitable time-scale"? who's willing to make sure things get fixed? pgpaMsk6vte8H.pgp Description: PGP signature
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Jakob Eriksson wrote: Maybe one could script something up so that a commit to tests CVS would not be *possible* without a confirmed test pass on Windows 95, NT, 2000 and XP. That would be very hard to do and mostly pointless for drivers. Ivan.
Enhancing winetest infrastructure [WAS: Wineconf agenda]
On Thursday 17 March 2005 10:33, Jakob Eriksson wrote: > Apart from all of Wine, I'm always interested in the conformance > testing. I believe it's crucial in speeding up Wines' development. > For each bug found, it is often a good idea to write an automatic > regression test. Yes, although that's a rather weary job, which no one enjoys doing :^/ > Maybe we can extend on the winetest.exe infrastructure. I would > like to see testing done in realtime [...] There's three issues here: o fast(er) update of CVS (or whatever filestore we're using). This would need either: * a super-charged Alexandre ;) or * a separate CVS tree in which developers can edit the wine/dlls/*/tests/ directory, or * give up on centralised/distributed testing architecture and switch to a personal testing environment. o fast(er) build of winetest.exe I originally argued for async. winetests and went as far as implementing this as part of WRT, so in principles this is already done. WRT worked based on the email notification of CVS updates. Builds, with minor changes, doesn't take long (using ccache), so you're probably looking 10-20 minutes turn around, with whatever delay the test-clients introduce. Though, without fixing the first issue, this doesn't help us much. o fast(er) running, through vmware platforms. Sure, this can be done, but its a distributed model, so everyone can chip in. Shouldn't be too difficult to achieve this. Just my 2-c worth :) Paul. pgpaeq8nAuEBd.pgp Description: PGP signature
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Maybe one could script something up so that a commit to tests CVS would not be *possible* without a confirmed test pass on Windows 95, NT, 2000 and XP. That'd be the best thing since sliced bread. regards, Jakob
Re: Enhancing winetest infrastructure [WAS: Wineconf agenda]
Paul Millar wrote: On Thursday 17 March 2005 10:33, Jakob Eriksson wrote: Apart from all of Wine, I'm always interested in the conformance testing. I believe it's crucial in speeding up Wines' development. For each bug found, it is often a good idea to write an automatic regression test. Yes, although that's a rather weary job, which no one enjoys doing :^/ I do! Maybe I have a condition, but I really love doing it! :-) There's three issues here: Yes. o fast(er) update of CVS (or whatever filestore we're using). This would need either: * a super-charged Alexandre ;) or * a separate CVS tree in which developers can edit the wine/dlls/*/tests/ directory, This makes sense. Winetest could actually be a separate project. I'm not saying it should be, mind you, just that conformance testing is interesting not only for Wine, but for Windows developers in general. So a separate CVS tree makes a great deal of sense, IMHO. Wine could import snapshots of the tests for its' conformance testing. This could speed up test development considerably, I imagine. or * give up on centralised/distributed testing architecture and switch to a personal testing environment. o fast(er) build of winetest.exe I originally argued for async. winetests and went as far as implementing this as part of WRT, so in principles this is already done. WRT worked based on the email notification of CVS updates. Builds, with minor changes, doesn't take long (using ccache), so you're probably looking 10-20 minutes turn around, with whatever delay the test-clients introduce. Though, without fixing the first issue, this doesn't help us much. o fast(er) running, through vmware platforms. Sure, this can be done, but its a distributed model, so everyone can chip in. Shouldn't be too difficult to achieve this. Sounds like fun, doesn't it? Test servers could register in the cluster worldwide. (Although I originally imagined a very centralized solution with a big server running vmware images.) Just my 2-c worth :) Are you coming to wineconf? regards, Jakob
