Re: CVS commit: src/distrib/sets/lists/xserver
On 5/20/11 10:28 AM, Izumi Tsutsui wrote: Module Name:src Committed By: tsutsui Date: Fri May 20 09:28:32 UTC 2011 Modified Files: src/distrib/sets/lists/xserver: md.zaurus Log Message: Add /etc/X11/xorg.conf. This should be in xetc, shouldn't it? Otherwise people will lose any local changes they do when they extract xserver.tgz. (I see other platforms installing xorg.conf-something from xserver.tgz, but I presume those are to be treated as examples.) -- Julio Merino / @jmmv
Re: CVS commit: src/tests/lib/libobjc
On 5/20/11 2:03 PM, Joerg Sonnenberger wrote: Module Name:src Committed By: joerg Date: Fri May 20 13:03:45 UTC 2011 Modified Files: src/tests/lib/libobjc: Makefile Log Message: Test case requires the GCC specific thread extension for Objective C, so only enable the test if GCC is present and disable other compiler. Tests should always run and report 'skipped' when they can't for whatever reason. This test should have a #ifdef around the code and fall back to a simple atf_tc_skip. -- Julio Merino / @jmmv
Re: CVS commit: src/tests/syscall
On Wed, May 18, 2011 at 12:30:24PM +, Christos Zoulas wrote: In article 4dd39547.9010...@julipedia.org, Julio Merino j...@julipedia.org wrote: On 5/18/11 4:15 AM, Christos Zoulas wrote: Module Name: src Committed By: christos Date: Wed May 18 03:15:12 UTC 2011 Modified Files: src/tests/syscall: t_pselect.c Log Message: Don't depend on the atf timeout stuff, do it ourselves. Why? Could atf do something better here? 1. It did not work. What units is the timeout anyway? I waited 2 seconds and 2 minutes and it did not fire. It's in seconds. The default is 300. You can override the timeout by setting the 'timeout' metadata property to a different value. 2. I need to do some cleanup if there is a timeout (kill the stuck process). How do I register a cleanup function? Use ATF_TC_WITH_CLEANUP and then provide an ATF_TC_CLEANUP function for your test case. Be aware that cleanup runs in a different process than the body, so the only way to transfer state is through a file in the current directory. 3. No matter how the timeout is done (unless you start a watcher process and kill -KILL the test process) it can break (masking signal mask, changing timers, catching signal). The timeout is enforced from atf-run, not from inside the test case; it is already using a helper process in a sense, so it should work. If it doesn't, it is a bug that I'd like to debug. -- Julio Merino
Re: CVS commit: src/distrib/sets/lists/xserver
Modified Files: src/distrib/sets/lists/xserver: md.zaurus Log Message: Add /etc/X11/xorg.conf. This should be in xetc, shouldn't it? Otherwise people will lose any local changes they do when they extract xserver.tgz. Hmm. I chose xorg.conf because there were very few NetBSD/zaurus users and there was no model specific hardware settings, so I thought less manual configuration was better to start Xserver after install. But if guys really claim we should keep a correct style in any case I'll rename it. I don't think it should be in xetc. It would cause much more unnecessary MACHINE dependent md.foo sets in releasedir. --- Izumi Tsutsui
Re: CVS commit: src/distrib/sets/lists/xserver
On 5/21/11 12:04 PM, Izumi Tsutsui wrote: Modified Files: src/distrib/sets/lists/xserver: md.zaurus Log Message: Add /etc/X11/xorg.conf. This should be in xetc, shouldn't it? Otherwise people will lose any local changes they do when they extract xserver.tgz. Hmm. I chose xorg.conf because there were very few NetBSD/zaurus users and there was no model specific hardware settings, so I thought less manual configuration was better to start Xserver after install. That's fine, but either the Xserver should just DTRT without a configuration file (like in does in other platforms), or the configuration file should really be treated as such. But if guys really claim we should keep a correct style in any case I'll rename it. Keeping it as xorg.conf is fine as long as it is in xetc.tgz. You don't want to screw up user-modified configuration files when they unpack xserver.tgz ;-) -- Julio Merino / @jmmv
Re: CVS commit: src/sys/arch/i386/stand
On Fri, May 20, 2011 at 10:29:56PM +, Joerg Sonnenberger wrote: Log Message: Disable integrated assembler for files that use .code16 or .code32 for now Would it have been better to do this with a level of indirection? So that it could be turned off by changing a global variable instead of having to edit all of the modified makefiles? David -- David Laight: da...@l8s.co.uk
Re: CVS commit: src/tests/syscall
In article 20110521083602.ga24...@netbsd.org, Julio Merino j...@netbsd.org wrote: On Wed, May 18, 2011 at 12:30:24PM +, Christos Zoulas wrote: In article 4dd39547.9010...@julipedia.org, Julio Merino j...@julipedia.org wrote: On 5/18/11 4:15 AM, Christos Zoulas wrote: Module Name: src Committed By: christos Date: Wed May 18 03:15:12 UTC 2011 Modified Files: src/tests/syscall: t_pselect.c Log Message: Don't depend on the atf timeout stuff, do it ourselves. Why? Could atf do something better here? 1. It did not work. What units is the timeout anyway? I waited 2 seconds and 2 minutes and it did not fire. It's in seconds. The default is 300. I think the unit is too long for modern hardware. Imaging having 300 tests needing 1 second timeouts. You can override the timeout by setting the 'timeout' metadata property to a different value. 2. I need to do some cleanup if there is a timeout (kill the stuck process). How do I register a cleanup function? Use ATF_TC_WITH_CLEANUP and then provide an ATF_TC_CLEANUP function for your test case. Be aware that cleanup runs in a different process than the body, so the only way to transfer state is through a file in the current directory. 3. No matter how the timeout is done (unless you start a watcher process and kill -KILL the test process) it can break (masking signal mask, changing timers, catching signal). The timeout is enforced from atf-run, not from inside the test case; it is already using a helper process in a sense, so it should work. If it doesn't, it is a bug that I'd like to debug. Revert the latest revision in the test and boot a kernel before the pselect changes, and see it getting stuck. christos
Re: CVS commit: src/tests/syscall
On 5/21/11 3:19 PM, Christos Zoulas wrote: In article20110521083602.ga24...@netbsd.org, Julio Merinoj...@netbsd.org wrote: 1. It did not work. What units is the timeout anyway? I waited 2 seconds and 2 minutes and it did not fire. It's in seconds. The default is 300. I think the unit is too long for modern hardware. Imaging having 300 tests needing 1 second timeouts. Yes, the timeout thing is broken. It should really be a specification of the test case size (e.g. 'small', 'large') and allow the user to define his timeout preferences for every class, because they will vary from machine to machine. That said, I don't see why you'd want to have 300 tests needing 1-second timeouts. If a test times out, it's broken, so the only thing I can deduce is that you'd have 300 broken tests ;-) 3. No matter how the timeout is done (unless you start a watcher process and kill -KILL the test process) it can break (masking signal mask, changing timers, catching signal). The timeout is enforced from atf-run, not from inside the test case; it is already using a helper process in a sense, so it should work. If it doesn't, it is a bug that I'd like to debug. Revert the latest revision in the test and boot a kernel before the pselect changes, and see it getting stuck. Any particular revision? I have a machine that hasn't been updated for at least two weeks; will that be enough? Thanks. -- Julio Merino / @jmmv
Re: CVS commit: src/tests/syscall
On May 21, 3:34pm, j...@netbsd.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | On 5/21/11 3:19 PM, Christos Zoulas wrote: | In article20110521083602.ga24...@netbsd.org, | Julio Merinoj...@netbsd.org wrote: | 1. It did not work. What units is the timeout anyway? I waited 2 seconds | and 2 minutes and it did not fire. | | It's in seconds. The default is 300. | | I think the unit is too long for modern hardware. Imaging having 300 tests | needing 1 second timeouts. | | Yes, the timeout thing is broken. It should really be a specification | of the test case size (e.g. 'small', 'large') and allow the user to | define his timeout preferences for every class, because they will vary | from machine to machine. | | That said, I don't see why you'd want to have 300 tests needing 1-second | timeouts. If a test times out, it's broken, so the only thing I can | deduce is that you'd have 300 broken tests ;-) | | 3. No matter how the timeout is done (unless you start a watcher process and | kill -KILL the test process) it can break (masking signal mask, changing | timers, catching signal). | | The timeout is enforced from atf-run, not from inside the test case; it is | already using a helper process in a sense, so it should work. If it doesn't, | it is a bug that I'd like to debug. | | Revert the latest revision in the test and boot a kernel before the pselect | changes, and see it getting stuck. | | Any particular revision? I have a machine that hasn't been updated for | at least two weeks; will that be enough? Oh sure. revision 1.1 christos
Re: CVS commit: src/tests/syscall
On Sat, May 21, 2011 at 03:34:22PM +0100, Julio Merino wrote: Yes, the timeout thing is broken. It should really be a specification of the test case size (e.g. 'small', 'large') and allow the user to define his timeout preferences for every class, because they will vary from machine to machine. Sure. I ran ATF on a machine with a 1Mhz CPU clock (this is a simulator, the real hardware is expected to run faster :). Lots of tests timed out just because on such hardware they can't complete in less than 5mn. -- Manuel Bouyer bou...@antioche.eu.org NetBSD: 26 ans d'experience feront toujours la difference --
Re: CVS commit: src/tests/syscall
On Sat, May 21, 2011 at 05:14:07PM +0200, Manuel Bouyer wrote: On Sat, May 21, 2011 at 03:34:22PM +0100, Julio Merino wrote: Yes, the timeout thing is broken. It should really be a specification of the test case size (e.g. 'small', 'large') and allow the user to define his timeout preferences for every class, because they will vary from machine to machine. Sure. I ran ATF on a machine with a 1Mhz CPU clock (this is a simulator, the real hardware is expected to run faster :). Lots of tests timed out just because on such hardware they can't complete in less than 5mn. Perhaps the timeout(s) should be settable units that default to 1 second. Them the timeouts can be scaled for very slow (or fast) systems. David -- David Laight: da...@l8s.co.uk
Re: CVS commit: src/tests/syscall
On May 21, 5:25pm, da...@l8s.co.uk (David Laight) wrote: -- Subject: Re: CVS commit: src/tests/syscall | On Sat, May 21, 2011 at 05:14:07PM +0200, Manuel Bouyer wrote: | On Sat, May 21, 2011 at 03:34:22PM +0100, Julio Merino wrote: | Yes, the timeout thing is broken. It should really be a | specification of the test case size (e.g. 'small', 'large') and | allow the user to define his timeout preferences for every class, | because they will vary from machine to machine. | | Sure. I ran ATF on a machine with a 1Mhz CPU clock (this is a | simulator, the real hardware is expected to run faster :). | Lots of tests timed out just because on such hardware they can't | complete in less than 5mn. | | Perhaps the timeout(s) should be settable units that default | to 1 second. | Them the timeouts can be scaled for very slow (or fast) systems. That should be easy to do since they are already strings. 1sec, 1msec, 1usec, 1nsec christos
Re: CVS commit: src/tests/syscall
On 5/21/11 5:25 PM, David Laight wrote: On Sat, May 21, 2011 at 05:14:07PM +0200, Manuel Bouyer wrote: On Sat, May 21, 2011 at 03:34:22PM +0100, Julio Merino wrote: Yes, the timeout thing is broken. It should really be a specification of the test case size (e.g. 'small', 'large') and allow the user to define his timeout preferences for every class, because they will vary from machine to machine. Sure. I ran ATF on a machine with a 1Mhz CPU clock (this is a simulator, the real hardware is expected to run faster :). Lots of tests timed out just because on such hardware they can't complete in less than 5mn. Perhaps the timeout(s) should be settable units that default to 1 second. Them the timeouts can be scaled for very slow (or fast) systems. Perhaps. But timeouts in a measure of time do not make sense in this context. There won't be any way to scale them in a reliable manner, so this will lead to flaky tests. And anyway, the user will be required to have control over how long these deadlines are. Again, there is no point in setting test timeouts to 1 second (except for very very rare cases). Timeouts are only used to kill tests that get stuck; nothing should rely on them. If there is many timeouts in your test suite (which will result in failed tests), something is broken and needs fixing. -- Julio Merino / @jmmv
Re: CVS commit: src/tests/syscall
On 05/21/11 15:44, Christos Zoulas wrote: On May 21, 3:34pm, j...@netbsd.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | On 5/21/11 3:19 PM, Christos Zoulas wrote: | Revert the latest revision in the test and boot a kernel before the pselect | changes, and see it getting stuck. | | Any particular revision? I have a machine that hasn't been updated for | at least two weeks; will that be enough? Oh sure. revision 1.1 Alright, so I just tried this. Downgraded the test to 1.1 and ensured that my kernel does not have the fix (sys_select.c:1.30 and sys_sig.c:1.32). And the timeout of 2 seconds works just fine. Just to make sure we are talking about the same: you were running the test program with atf-run; correct? If yes, what platform was this in? (I'm testing with amd64.) Otherwise, this is expected behavior: tests are not supposed to be executed without atf-run.
Re: CVS commit: src/tests/syscall
On May 21, 9:26pm, j...@julipedia.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | On 05/21/11 15:44, Christos Zoulas wrote: | On May 21, 3:34pm, j...@netbsd.org (Julio Merino) wrote: | -- Subject: Re: CVS commit: src/tests/syscall | | | On 5/21/11 3:19 PM, Christos Zoulas wrote: | | Revert the latest revision in the test and boot a kernel before the pselect | | changes, and see it getting stuck. | | | | Any particular revision? I have a machine that hasn't been updated for | | at least two weeks; will that be enough? | | Oh sure. revision 1.1 | | Alright, so I just tried this. Downgraded the test to 1.1 and ensured | that my kernel does not have the fix (sys_select.c:1.30 and sys_sig.c:1.32). | | And the timeout of 2 seconds works just fine. | | Just to make sure we are talking about the same: you were running the | test program with atf-run; correct? If yes, what platform was this in? | (I'm testing with amd64.) Otherwise, this is expected behavior: tests | are not supposed to be executed without atf-run. No, I am running it directly. Programs could warn if they figured out that they are not running under atf-run. Print something like: not running under atf-run, timeouts will not work. Also it would be nice if programs when invoked with -l or something they would list all their tests, and if invoked with -a they would run all the tests. christos
Re: CVS commit: src/sys/arch/i386/stand
On Sat, May 21, 2011 at 01:01:52PM +0100, David Laight wrote: On Fri, May 20, 2011 at 10:29:56PM +, Joerg Sonnenberger wrote: Log Message: Disable integrated assembler for files that use .code16 or .code32 for now Would it have been better to do this with a level of indirection? So that it could be turned off by changing a global variable instead of having to edit all of the modified makefiles? I didn't want to mess with the structure of i386/stand at this point. There is already a lot of redundancy and I hope this instance will go away in the not so far future. Joerg
Re: CVS commit: src/tests/syscall
On 05/21/11 21:58, Christos Zoulas wrote: On May 21, 9:26pm, j...@julipedia.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | Just to make sure we are talking about the same: you were running the | test program with atf-run; correct? If yes, what platform was this in? | (I'm testing with amd64.) Otherwise, this is expected behavior: tests | are not supposed to be executed without atf-run. No, I am running it directly. Programs could warn if they figured out that they are not running under atf-run. Print something like: not running under atf-run, timeouts will not work. Sounds like a good idea. Note that there is *many* more things that will not work. The test case will dump garbage in the current directory and it won't be cleaned, the environment won't be sanitized, child processes won't be killed... Also it would be nice if programs when invoked with -l or something they would list all their tests, and if That already works :-) The output format is gly though invoked with -a they would run all the tests. Tests used to do exactly that more than a year ago, but it turned out to be a bad idea... The test programs were extremely complex: they had to implement all the test case isolation (work directory, timeouts, etc.) themselves, and that meant having to implement the same thing in C, C++ and shell. Also, and this is what bugged most people: debugging the test cases was almost impossible due to all the black magic that was involved in them. The test cases would delete temporary files on failure and gdb was unusable. As a result, I moved all the isolation logic into atf-run and turned the test programs into a 'dumb' frontend for the test cases. Test programs now don't do much more than just running test case bodies and exposing a consistent interface, so they really should not be used directly unless you need to debug a particular test case. Oh, and the whole system works much, much faster ;-P
Re: CVS commit: src/tests/syscall
On May 21, 10:31pm, j...@julipedia.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | Sounds like a good idea. Note that there is *many* more things that | will not work. The test case will dump garbage in the current directory | and it won't be cleaned, the environment won't be sanitized, child | processes won't be killed... It could mention all of them. | | Also it would be nice if programs | when invoked with -l or something they would list all their tests, and if | | That already works :-) The output format is gly though Yes, perhaps use a different flag to just list the names. | invoked with -a they would run all the tests. | | Tests used to do exactly that more than a year ago, but it turned out to | be a bad idea... | | The test programs were extremely complex: they had to implement all the | test case isolation (work directory, timeouts, etc.) themselves, and | that meant having to implement the same thing in C, C++ and shell. I don't see why. You just put it all in a shared library written in c and used from c++. Shell you have already. | Also, and this is what bugged most people: debugging the test cases was | almost impossible due to all the black magic that was involved in them. | The test cases would delete temporary files on failure and gdb was | unusable. You can have a flag to disable the setup/cleanup. | As a result, I moved all the isolation logic into atf-run and turned the | test programs into a 'dumb' frontend for the test cases. Test programs | now don't do much more than just running test case bodies and exposing a | consistent interface, so they really should not be used directly unless | you need to debug a particular test case. Oh, and the whole system | works much, much faster ;-P Well, I've never ran atf-run :-) and I've ran individual tests multiple times. christos
Re: CVS commit: src/tests/syscall
On 5/22/11 12:18 AM, Christos Zoulas wrote: On May 21, 10:31pm, j...@julipedia.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | Sounds like a good idea. Note that there is *many* more things that | will not work. The test case will dump garbage in the current directory | and it won't be cleaned, the environment won't be sanitized, child | processes won't be killed... It could mention all of them. atf-test-case(4) contains a description of these. (Yeah, it could be improved.) | Tests used to do exactly that more than a year ago, but it turned out to | be a bad idea... | | The test programs were extremely complex: they had to implement all the | test case isolation (work directory, timeouts, etc.) themselves, and | that meant having to implement the same thing in C, C++ and shell. I don't see why. You just put it all in a shared library written in c and used from c++. Shell you have already. That's what was done. But in order to enforce the timeouts et. al. this library had to fork() on every test case. This was really, really ugly (and slow) in shell. It was also hard to keep the 3 implementations (c, c++, shell) in sync. I'm not saying it's not possible, just that it was ugly _in the past_. We can revisit this later. The renewed runtime I'm working on (atf v2, aka kyua) will support different test program styles, so we should be able to easily re-experiment with this approach.
Re: CVS commit: src/tests/syscall
On May 22, 12:28am, j...@julipedia.org (Julio Merino) wrote: -- Subject: Re: CVS commit: src/tests/syscall | atf-test-case(4) contains a description of these. (Yeah, it could be | improved.) Good. | I don't see why. You just put it all in a shared library written in c and | used from c++. Shell you have already. | | That's what was done. But in order to enforce the timeouts et. al. this | library had to fork() on every test case. This was really, really ugly | (and slow) in shell. It was also hard to keep the 3 implementations (c, | c++, shell) in sync. You should not have to fork if you don't set a timeout. Or you can fork just one killer and have it timeout in 2x the last registered timeout and exit. It listens to a socket for pid's and timeouts; if the socket is still useable after the timeout it kills. | I'm not saying it's not possible, just that it was ugly _in the past_. | We can revisit this later. The renewed runtime I'm working on (atf v2, | aka kyua) will support different test program styles, so we should be | able to easily re-experiment with this approach. Yes, the implementation needs to be done carefully. christos
