RE: Test timeouts
From: Alexandre Julliard > of course the testbot having a 5 minutes timeout makes this worse Yeah, I don't remember the reasoning behind that choice. It probably makes more sense to use the same timeout as winetest. Thanks for making your position clear before I spent time on implementing stuff, I don't necessarily agree with your position but I guess that's part of the Wine development process :-). Greg.
Re: Test timeouts
Greg Geldorp writes: >> From: Alexandre Julliard >> >> I think it would be better to make the test detect such an environment >> and have it skip some things if it finds that it's taking too long. > > I guess that's what we currently have... In effect, if it takes too long, > the final tests are skipped. I was hoping we could do better than that. The difference is that a test killed by a timeout is considered an error, because it's not supposed to happen. If the test knows that it can happen it should take measures to prevent it. I don't think increasing the timeout is a solution, because there's no clear value for what the maximum should be, so to be safe you have to make it way too long. There are already cases where I apply patches without waiting for the testbot to check them, because it's busy with some other test that's timing out (of course the testbot having a 5 minutes timeout makes this worse). We really need to enforce that tests finish in a reasonable time. -- Alexandre Julliard julli...@winehq.org
RE: Test timeouts
> From: Alexandre Julliard > > I think it would be better to make the test detect such an environment > and have it skip some things if it finds that it's taking too long. I guess that's what we currently have... In effect, if it takes too long, the final tests are skipped. I was hoping we could do better than that. Greg.
Re: Test timeouts
Greg Geldorp writes: > Although the number of timeouts in msi tests has dropped considerably after > Hans decided to do something about them, they're unfortunately not > completely gone. It seems that the remaining timeouts have a strong > dependency on the environment. It's possible to tweak the environment > (disable system restore points, for TestBot I found it makes a difference > on which physical disk the virtual disk files are located) but the goal > should be to have the tests run consistently on whatever the environment > happens to be. Perhaps it's time to accept that some tests in some > environments can simply take longer than the current winetest.exe timeout > value of 2 minutes. I think it would be better to make the test detect such an environment and have it skip some things if it finds that it's taking too long. IOW it would be the responsibility of the test itself to keep its run time under 2 minutes. -- Alexandre Julliard julli...@winehq.org
Re: Test timeouts
On Wed, 2011-01-05 at 14:01 +0100, wy...@volny.cz wrote: > s there is an activity to make (not only) WylBOT green and happy > ;c) i can add more info, based on WyldBOT's install documentation i > looked right now and found there: > > * Turn off system restore > > So longer time out may not help my BOTs, as it's already turned off > on all of the BOTs and the HW isn't so weak/slow, i guess... Greg was talking about timeouts caused by msi tests simply taking too long to run on some configurations. This problem can be addressed by increasing timeouts. There are other sources of timeouts, such as dialogs popping up, which cannot be fixed that way. Please report those directly to me or create a bug report.
Test timeouts
> > ... It seems that the remaining > timeouts have a strong > dependency on the environment. It's possible to > tweak the environment > (disable system restore points, for TestBot I > found it makes a difference > on which physical disk the virtual disk files are > located) > > ... some environments can simply take longer than the > current winetest.exe timeout > value of 2 minutes. > As there is an activity to make (not only) WylBOT green and happy ;c) i can add more info, based on WyldBOT's install documentation i looked right now and found there: * Turn off system restore So longer time out may not help my BOTs, as it's already turned off on all of the BOTs and the HW isn't so weak/slow, i guess... W.
Test timeouts
Although the number of timeouts in msi tests has dropped considerably after Hans decided to do something about them, they're unfortunately not completely gone. It seems that the remaining timeouts have a strong dependency on the environment. It's possible to tweak the environment (disable system restore points, for TestBot I found it makes a difference on which physical disk the virtual disk files are located) but the goal should be to have the tests run consistently on whatever the environment happens to be. Perhaps it's time to accept that some tests in some environments can simply take longer than the current winetest.exe timeout value of 2 minutes. I can think of 3 possible solutions: 1) Simply increase the timeout value in winetest.exe. Extremely simple to implement, but perhaps not very elegant. 2) Pass the timeout value as a command line parameter to winetest.exe. This has the advantage that the timeout can be adjusted for the environment (creating a system restore point on an older machine with lots of stuff installed will take longer than creating a system restore point on a fresh high-end machine), it does put the responsibility for determining a suitable timeout on the shoulders of the tester. That's probably no problem for regular testers, but we can't expect the occasional tester to do this. 3) Have a variable timeout per testset. One possible way to implement this would be to have the test executables recognize an additional command line parameter "--timeouts". When called with this parameter, the test executable tries to load a resource string containing a list of testset-specific timeout values, for msi it could be something like "action:180;install:180;msi:300". Testsets not included in the resource string would get the default timeout of 120 sec, if the resource string is not present all testsets would get that default. I believe this can be implemented in wine/test.h, so the only change required to the tests themselves would be to add the resource string to tests that require a different timeout. Instead of calling the test executable with the "--list" command line parameter winetest.exe would call it with "--timeouts", which would produce a list of the testssets with their timeouts. The advantage of this method is that it's quite flexible. It is a bit more work to implement. Personally, I have a slight preference for solution 3) but I can live with the others too. Obviously I'm willing to implement whatever solution we choose. Greg.
Re: Solution to msi/install test timeouts
On Fri, 2008-08-29 at 11:01 +0200, Paul Vriens wrote: > Hi, > > I've been experimenting on my XP and Vista boxes and the only way to make the > tests run fast is by stopping 'System Restore'. > > It makes sense that this has a huge impact on performance, especially with > the > big number of installations we are doing in these tests. > > Could people that are currently seeing timeouts for their boxes on > test.winehq.org check if 'System Restore' is indeed activated? > Both of my machines have System Restore enabled.
Solution to msi/install test timeouts
Hi, I've been experimenting on my XP and Vista boxes and the only way to make the tests run fast is by stopping 'System Restore'. It makes sense that this has a huge impact on performance, especially with the big number of installations we are doing in these tests. Could people that are currently seeing timeouts for their boxes on test.winehq.org check if 'System Restore' is indeed activated? -- Cheers, Paul.
urlmon:protocol test timeouts
Hi Jacek, The following patch causes timeouts for all platforms, can you have a look at it? commit 210854edf2e841af675d541d29f2b90cd2625d02 Author: Jacek Caban <[EMAIL PROTECTED]> Date: Thu Jan 31 18:18:49 2008 +0100 urlmon: Added more protocol tests. http://test.winehq.org/data/156dad05c08cd921288aedf1b281ff23844a1692/ Thanks, James Hawkins
