bug#48113: Module suggestion: timeout
Just to mention further evidence supporting Simon's final idea of "choosing the default timeout setting is difficult". This report recently came into bug-dejagnu precisely about different timeouts being needed on different systems: https://lists.gnu.org/archive/html/bug-dejagnu/2021-05/msg00011.html And the replies by the maintainer reject the idea of a single timeout. No action required, just thought it was interesting that the same general issue came up elsewhere. -k
bug#48113: Module suggestion: timeout
severity 48113 wishlist retitle 48113 Self-test timeout functionality thanks Karl Berry writes: > What do bug-automake people think? > > For myself, I have no objection to sprinkling timeout commands through > the Automake test infrastructure wherever appropriate. It's not ever > going to rise to the top of my own list of things to do, though, so if > it's going to happen, you or someone will have to write the patch. > > Of course I don't speak for Jim, but from what he's said in the past, > I suspect he is in a similar situation. Thanks for confirming that it isn't an obviously bad idea. Tagging this as a wishlist bug, and improving the bug title a bit. > No one else has come forward to work on Automake, despite my plea > (https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html), > so I guess that's where we are. Ah right -- I'll see if I can help in any way, but will respond separately. > the functionality could be opt-in initially, > > Certainly. > > and then after a few years become the default behaviour. > > Personally, I think it should be opt-in forever. People could easily > have test suites that need to run for days. I prefer not to > unnecessarily break compatibility. Yes, I probably agree with this -- chosing the default timeout setting is difficult as it will most likely just cause problems for some people with long-running tests, and not solve the initial problem for some other people (that have small projects and a timeout of 5 minutes is what you want). Better leave this up the each project to decide. /Simon signature.asc Description: PGP signature
bug#48113: Module suggestion: timeout
What do bug-automake people think? For myself, I have no objection to sprinkling timeout commands through the Automake test infrastructure wherever appropriate. It's not ever going to rise to the top of my own list of things to do, though, so if it's going to happen, you or someone will have to write the patch. Of course I don't speak for Jim, but from what he's said in the past, I suspect he is in a similar situation. No one else has come forward to work on Automake, despite my plea (https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html), so I guess that's where we are. the functionality could be opt-in initially, Certainly. and then after a few years become the default behaviour. Personally, I think it should be opt-in forever. People could easily have test suites that need to run for days. I prefer not to unnecessarily break compatibility. Thanks, Karl
bug#48113: Module suggestion: timeout
Adding timeout to all build operations automatically is probably too much and too intrusive. But in any case, it should be easy to add such a timeout to certain build operations (e.g. downloading, running a compiler for which the halting problem cannot be solved, testing a production binary, ...). Am Fr., 30. Apr. 2021 um 11:43 Uhr schrieb Simon Josefsson < si...@josefsson.org>: > Marc Nieper-Wißkirchen writes: > > > Moreover, use cases for a baked-in timeout are not restricted to tests. > For > > example, I may want to restrict the build time of certain components in > > situations where a logical error may lead to infinite build times (a > simple > > example is that of a Scheme compiler used as a build tool; thanks to > > Turing-completeness of Scheme macros, such a build may not terminate). > > This makes me believe even stronger that the functionality ought to be > provided by automake natively -- it seems the desired functionality is > not only timeouts for self-tests but timeouts for all operations. > > Implementing this for self-tests in automake would probably be quite > simple, but implementing it for all operations is probably more > complicated. Maybe it should be two separate features. I have wanted > timeouts for self-tests but rarely for building. If the second is more > complicated to implement, maybe starting with the first will be > sufficient and useful. > > /Simon >
bug#48113: Module suggestion: timeout
Marc Nieper-Wißkirchen writes: > Moreover, use cases for a baked-in timeout are not restricted to tests. For > example, I may want to restrict the build time of certain components in > situations where a logical error may lead to infinite build times (a simple > example is that of a Scheme compiler used as a build tool; thanks to > Turing-completeness of Scheme macros, such a build may not terminate). This makes me believe even stronger that the functionality ought to be provided by automake natively -- it seems the desired functionality is not only timeouts for self-tests but timeouts for all operations. Implementing this for self-tests in automake would probably be quite simple, but implementing it for all operations is probably more complicated. Maybe it should be two separate features. I have wanted timeouts for self-tests but rarely for building. If the second is more complicated to implement, maybe starting with the first will be sufficient and useful. /Simon signature.asc Description: PGP signature
bug#48113: Module suggestion: timeout
Dear Bruno, dear Simon, thank you for your replies. I understand that valgrind-tests and the proposed "timeout-tests" solution are not completely equivalent. Nevertheless, I still think that some timeout functionality provided by Gnulib would be useful. Bruno's solution ** #if HAVE_DECL_ALARM /* Declare failure if test takes too long, by using default abort caused by SIGALRM. */ int alarm_value = 600; signal (SIGALRM, SIG_DFL); alarm (alarm_value); #endif ** works for unit tests that have been written specifically for the project. It doesn't help, though, if I want to test (production) binaries that are to be installed because I generally don't want testing code inside them. Moreover, use cases for a baked-in timeout are not restricted to tests. For example, I may want to restrict the build time of certain components in situations where a logical error may lead to infinite build times (a simple example is that of a Scheme compiler used as a build tool; thanks to Turing-completeness of Scheme macros, such a build may not terminate). Marc Am Fr., 30. Apr. 2021 um 11:24 Uhr schrieb Simon Josefsson < si...@josefsson.org>: > Bruno Haible writes: > > > So, I don't think the "let's treat timeout like valgrind" approach is > going > > to work. Instead, you need to design a way to deal with timeouts, > independently. > > Hi! I think Marc's request for functionality to introduce timeouts for > self-tests is a good one. However I reach the same conclusion as Bruno, > that having a module like valgrind-tests is probably not the best way to > solve it. To me, having a timeout seems like an essential feature of a > self-test framework. I know automake isn't primarily a self-test > framework, but it has concepts for it and the test framework has been > improved significantly over the years, so I think adding a timeout > functionality to automake makes sense. What do bug-automake people > think? > > The functionality could be conditioned on the coreutils 'timeout' tool, > and if that tool exists, and appears to work, running all self-tests > under that tool could be done automatically. The default self-test > timeout be quite generous (say 17 hours?) but it should be easy to > modify both by end-user and project developer. If we want to be > conservative, the functionality could be opt-in initially, and then > after a few years become the default behaviour. > > Thoughts? > > /Simon >
bug#48113: Module suggestion: timeout
Bruno Haible writes: > So, I don't think the "let's treat timeout like valgrind" approach is going > to work. Instead, you need to design a way to deal with timeouts, > independently. Hi! I think Marc's request for functionality to introduce timeouts for self-tests is a good one. However I reach the same conclusion as Bruno, that having a module like valgrind-tests is probably not the best way to solve it. To me, having a timeout seems like an essential feature of a self-test framework. I know automake isn't primarily a self-test framework, but it has concepts for it and the test framework has been improved significantly over the years, so I think adding a timeout functionality to automake makes sense. What do bug-automake people think? The functionality could be conditioned on the coreutils 'timeout' tool, and if that tool exists, and appears to work, running all self-tests under that tool could be done automatically. The default self-test timeout be quite generous (say 17 hours?) but it should be easy to modify both by end-user and project developer. If we want to be conservative, the functionality could be opt-in initially, and then after a few years become the default behaviour. Thoughts? /Simon signature.asc Description: PGP signature
