Re: [O] Temp files from testing are permanent...
Achim Gratz strom...@nexgo.de writes: Eric Schulte eric.schu...@gmx.com writes: It will be up to the authors of individual tests to remove tangled and exported files. Ideally we can patch each test to clean up after itself. Perhaps we should provide a test macro which accepts a list of file names and optionally removes them if the test exists successfully. e.g., (org-test-with-cleanup '(exported.html tangled.sh etc...) ...test body...) Currently those files would have inpredictable names AFAICS, which would defeat that purpose. I should have been more clear. I'm thinking that this would be a macro used /within/ unit tests so that testers could specify what files will be created (test writers should be able to predict the file names created by their tests) and then the macro will handle cleanup. Here's one implementation of such a macro (defmacro org-test-with-cleanup (files rest body) If BODY executes successfully delete FILES and return results of BODY. (declare (indent 1)) `(let ((result (progn ,@body))) (mapc (lambda (file) (when (file-exists-p file) (delete-file file))) files) result)) I'm not sure where to do this, but it occurs to me that it would be easier if those files only had a unique prefix that didn't change for each invocation of the test suite (a timestamp would be OK, so it is easier to see which files you're looking at). If the files always had the same name, you would have to make sure that there was no collision, either with existing files or results from earlier tests, so that would be no good. If it is easier to always have the same names for the files, one could just make the test directory name unique. I think it is useful for debugging to be able to keep a few generation of tests around. I do like the idea of a single directory in which all output files may be collected. The only potential downside I see for this is that files will be generated both from within org files in the testing/examples directory as well as temporary files. Even if the above is implemented this sounds like a good safeguard. Safeguards are in place in my Makefile fork. Regards, Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Temp files from testing are permanent...
Eric Schulte eric.schu...@gmx.com writes: I should have been more clear. I'm thinking that this would be a macro used /within/ unit tests so that testers could specify what files will be created (test writers should be able to predict the file names created by their tests) and then the macro will handle cleanup. I'm sitting on the fence with this. Running the tests from the Makefile it would probably be more difficult to ensure that one could keep the files when the tests were trying to clean up after themselves (as some option would need to be injected into the test invocation and/or a different test command would need to be called). I do like the idea of a single directory in which all output files may be collected. The only potential downside I see for this is that files will be generated both from within org files in the testing/examples directory as well as temporary files. The temporary files could be in a sub-directory... or each test (group) could have their own sub-directory. Whatever the organisation, there should be a single directory which, if recursively removed, gets rid of all files created by the test run. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Waldorf MIDI Implementation additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
Re: [O] Temp files from testing are permanent...
FTR, I'm just commenting based on experience with a testing harness for a completely unrelated piece of software. Achim Gratz strom...@nexgo.de writes: Eric Schulte eric.schu...@gmx.com writes: I should have been more clear. I'm thinking that this would be a macro used /within/ unit tests so that testers could specify what files will be created (test writers should be able to predict the file names created by their tests) and then the macro will handle cleanup. Test writers can predict/choose the file names created by their tests but they cannot predict the file names creates by other test writers' tests (or their own tests written two weeks ago ;-). Unless there is some naming policy that is strictly adhered to, the chance of collisions remains. One approach that has worked for me is to have tests create their outputs in a directory named after the test. So if I have a test implemented in test.el, it would create all outputs in test.out/, for example. As I put all my tests and their inputs in a tests/ directory (and nothing else), I can just `rm -rf *.out` there to clean up. I'm sitting on the fence with this. Running the tests from the Makefile it would probably be more difficult to ensure that one could keep the files when the tests were trying to clean up after themselves (as some option would need to be injected into the test invocation and/or a different test command would need to be called). Personally, I prefer to have `make clean` take care of cleaning up my test outputs. I control when it gets run, can move valuable stuff out of the way before doing so and none of the tests have to bother with conditionalized clean up. I do like the idea of a single directory in which all output files may be collected. The only potential downside I see for this is that files will be generated both from within org files in the testing/examples directory as well as temporary files. The temporary files could be in a sub-directory... or each test (group) could have their own sub-directory. Whatever the organisation, there should be a single directory which, if recursively removed, gets rid of all files created by the test run. See above. Hope this helps, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Re: [O] Temp files from testing are permanent...
Achim Gratz strom...@nexgo.de writes: Olaf Meeuwissen olaf.meeuwis...@avasys.jp writes: Successful tests can clean up after themselves but failed tests should not so you can debug. The decision to remove these files should be left to whoever runs the test suite. That implies that even successful tests don't really have to bother cleaning up after themselves. As hinted at in another post yesterday, I did browse the test suite code briefly. The tests try to clean up after themselves, sort-of Sorry to come late to this discussion. The tests should clean up all temporary files and buffers left during their execution. There has been some work on this in the last month or two so hopefully the situation has improved. — the trouble is that they leave tangled files and results produced during suceesful tests. It will be up to the authors of individual tests to remove tangled and exported files. Ideally we can patch each test to clean up after itself. Perhaps we should provide a test macro which accepts a list of file names and optionally removes them if the test exists successfully. e.g., (org-test-with-cleanup '(exported.html tangled.sh etc...) ...test body...) If you use `make check` to run the test suite, you can easily set TMPDIR via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable is taken into account when making unique file names). Something like Yes, that's about the same conclusion I also reached. Not sure when I'll have time to implement and test it. Even if the above is implemented this sounds like a good safeguard. Thanks, Regards, Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Temp files from testing are permanent...
Eric Schulte eric.schu...@gmx.com writes: It will be up to the authors of individual tests to remove tangled and exported files. Ideally we can patch each test to clean up after itself. Perhaps we should provide a test macro which accepts a list of file names and optionally removes them if the test exists successfully. e.g., (org-test-with-cleanup '(exported.html tangled.sh etc...) ...test body...) Currently those files would have inpredictable names AFAICS, which would defeat that purpose. I'm not sure where to do this, but it occurs to me that it would be easier if those files only had a unique prefix that didn't change for each invocation of the test suite (a timestamp would be OK, so it is easier to see which files you're looking at). If the files always had the same name, you would have to make sure that there was no collision, either with existing files or results from earlier tests, so that would be no good. If it is easier to always have the same names for the files, one could just make the test directory name unique. I think it is useful for debugging to be able to keep a few generation of tests around. Even if the above is implemented this sounds like a good safeguard. Safeguards are in place in my Makefile fork. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Waldorf MIDI Implementation additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
Re: [O] Temp files from testing are permanent...
Olaf Meeuwissen olaf.meeuwis...@avasys.jp writes: Successful tests can clean up after themselves but failed tests should not so you can debug. The decision to remove these files should be left to whoever runs the test suite. That implies that even successful tests don't really have to bother cleaning up after themselves. As hinted at in another post yesterday, I did browse the test suite code briefly. The tests try to clean up after themselves, sort-of — the trouble is that they leave tangled files and results produced during suceesful tests. If you use `make check` to run the test suite, you can easily set TMPDIR via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable is taken into account when making unique file names). Something like Yes, that's about the same conclusion I also reached. Not sure when I'll have time to implement and test it. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Re: [O] Temp files from testing are permanent...
Achim Gratz strom...@nexgo.de writes: Barring any setup or options I might have missed, it looks like I will have to do two things: first, set TMPDIR to a newly created directory during the test runs and second, remove that directory after the test (with an option to keep it for inspection and removing only with clean). Done and pushed to my Makefile fork on repo.or.cz. Temporary test files automatically vaporize when the test suite successfully runs and are left for inspection otherwise. If you need to keep them regardless of the test result, define TEST_NO_AUTOCLEAN non-empty. A new target cleantest can be used to remove the temporary test directory and all its content whenever necessary, this is also done when doing a cleanall. The location of the temporary test directory follows $TMPDIR and can be overriden in local.mk via variable testdir. TMPDIR, if not already present in the environment, will default to /tmp, which incidentally should resolve a bug in my tricky build environment at work. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Temp files from testing are permanent...
Olaf Meeuwissen olaf.meeuwis...@avasys.jp writes: If running `make check` (or similar) creates these files, `make clean` (or similar) should clean them up. I was asking that question to decide whether I do need to extend my Makefile fork to handle the cleanup or if the testsuite needs to be called differently to avoid leaving these stale files. If you'd take a look, you can see that the files are obviously made with a function that ensures they're unique, most likely make-temp-file. If so, it would seem logical that the testsuite should remove them after each test (since it is the only instance to know the complete filename). It would also seem logical that for debugging purposes one could leave the files around. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf rackAttack V1.04R1: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Temp files from testing are permanent...
On Wed, Feb 15, 2012 at 11:11 AM, Achim Gratz strom...@nexgo.de wrote: Olaf Meeuwissen olaf.meeuwis...@avasys.jp writes: If running `make check` (or similar) creates these files, `make clean` (or similar) should clean them up. I was asking that question to decide whether I do need to extend my Makefile fork to handle the cleanup or if the testsuite needs to be called differently to avoid leaving these stale files. ... It would also seem logical that for debugging purposes one could leave the files around. If created with make-temp-file, and created in the system-configured $TMPDIR directory, I would urge (wearing my sysadmin hat) that they get purged at the end of the test run, unless told to do otherwise. If created in the build/test directory, then either make clean or immediate purge seems reasonable. Files created in the system $TMPDIR are not meant (caution: purist view) to remain beyond the execution of the program (or set of programs). Some OS variants are set up to purge the $TMPDIR on reboot, login, or at other times, and some even store it in a virtual memory backed filesystem. When performing as a sysadmin, finding that an application has littered a (usually) limited system resource such as the system $TMPDIR with files that are no longer useful is a minor irritant at best, to a crash-inducing resource consumer at worst. Just my $0.02 if you are taking donations. Brian
Re: [O] Temp files from testing are permanent...
Brian Wightman midlife...@wightmanfam.org writes: Files created in the system $TMPDIR are not meant (caution: purist view) to remain beyond the execution of the program (or set of programs). You are preaching to the choir... :-) I'm still looking for some wisdom regarding the actual testsuite setup in orgmode, though. We have org-test-with-temp-text-in-file in org-test.el, which supposedly cleans up after itself. Other tests aren't as careful, specifically files with the following prefixes: awk-, C-bin-, C-src-, emacs, fortran-bin-, fortan-src-, maxima-, octave-, org-test (seems to be an HTML export from a temporary org file that does get removed), python-, scor and sh-. Barring any setup or options I might have missed, it looks like I will have to do two things: first, set TMPDIR to a newly created directory during the test runs and second, remove that directory after the test (with an option to keep it for inspection and removing only with clean). Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Re: [O] Temp files from testing are permanent...
Achim Gratz strom...@nexgo.de writes: Olaf Meeuwissen olaf.meeuwis...@avasys.jp writes: If running `make check` (or similar) creates these files, `make clean` (or similar) should clean them up. I was asking that question to decide whether I do need to extend my Makefile fork to handle the cleanup or if the testsuite needs to be called differently to avoid leaving these stale files. If you'd take a look, you can see that the files are obviously made with a function that ensures they're unique, most likely make-temp-file. If so, it would seem logical that the testsuite should remove them after each test (since it is the only instance to know the complete filename). It would also seem logical that for debugging purposes one could leave the files around. Successful tests can clean up after themselves but failed tests should not so you can debug. The decision to remove these files should be left to whoever runs the test suite. That implies that even successful tests don't really have to bother cleaning up after themselves. I normally instruct my tests to create files in a dedicated directory with a fixed name somewhere below $(top_builddir). Then a `make clean` can just remove the whole directory. If you use `make check` to run the test suite, you can easily set TMPDIR via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable is taken into account when making unique file names). Something like TESTS_ENVIRONMENT = TMPDIR=$(builddir)/test-outputs and then add a clean-local target like clean-local: -rm -rf $(builddir)/test-outputs See [[info:automake-1.11#Simple%20Tests][info:automake-1.11#Simple Tests]] and [[info:automake-1.11#Extending]] for details. Hope this helps, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
[O] Temp files from testing are permanent...
I've started to always test each build. It seems that the temporary files from testing (they end up in /tmp) are not actually removed after testing and continue to pile up. Is there an option that I might have missed that prevents these files from being removed or should cleaning those files be added to the Makefile? Im starting the test suite with /usr/local/bin/emacs -batch -u myself -L lisp/ \ --eval '(defconst org-release 7.8.03-Test)' \ -l testing/org-test.el \ -eval (setq org-confirm-babel-evaluate nil) \ -f org-test-run-batch-tests Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf rackAttack V1.04R1: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Temp files from testing are permanent...
Achim Gratz strom...@nexgo.de writes: I've started to always test each build. It seems that the temporary files from testing (they end up in /tmp) are not actually removed after testing and continue to pile up. Is there an option that I might have missed that prevents these files from being removed or should cleaning those files be added to the Makefile? Im starting the test suite with If running `make check` (or similar) creates these files, `make clean` (or similar) should clean them up. See [[info:automake-1.11#Clean]] for details. /usr/local/bin/emacs -batch -u myself -L lisp/ \ --eval '(defconst org-release 7.8.03-Test)' \ -l testing/org-test.el \ -eval (setq org-confirm-babel-evaluate nil) \ -f org-test-run-batch-tests Hope this helps, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
