Re: [fricas-devel] Regression testing
On 27/09/16 18:55, Waldek Hebisch wrote: Since sometimes there are good reasons for irregularity I do not insist on rigid structure. But I would like to avoid irregularities which are just due to slightly different preferences. Yes, its a difficult dilemma. I really appreciate lack of restrictions when writing code (and your help improving it), not so much so when trying to understand other peoples code. But it was _much_ easier to add tests to 'bugs2016.input'. OK, the next tests I write I will use an .input file. Then I can make more informed comments on compiler vs. interpreter for tests. Martin B -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
Re: [fricas-devel] Regression testing
Martin Baker wrote: > > On 25/09/16 23:31, Waldek Hebisch wrote: > > > Concerning organization of tests, I do not buy argument > > that "it must be in the same file" > > OK, how about having an additional directory like src/algebra where > there is a test file corresponding to every (eventually) spad file in > src/algebra. > I think it would really help if the test code and source code were 1:1. Well, integrator is split into several files. OTOH "new" integration tests are in 'integ.input'. They re-use a few helper routines and in general integration tests are related so it makes sense to keep them in a single file. There are also older tests, which do not use test machinery, so they are in separate files. In other words there are technical and historical reasons to deviate from 1:1 correspondence. If you want rough correspondence, then this is already happening for example guessing package is mostly in mantepse.spad, tests are in mantepse.input. But there are also deviations. > > What is in which > > file matters mostlty for Makefiles and grep searches. > > Putting things different than source code means a lot > > of false hits from grep, which is a significant > > distraction. Also Makefile may get unnecessarly complicated. > > When doing "quick and dirty" fixes people will go > > directly to relevant function -- if you add something > > 20 lines below it may be as easily missed as something > > in completely different file. > > I think it is important to design these things from the point of view of > someone new to FriCAS. Imagine, if you can, that I was new to FriCAS, > that I wanted to contribute to the code, that I was not familiar with > the test framework and that I believed in test driven development. > > Would I know where to put the test code? Would I have to resort to > searching the source to find existing tests? Well, most important ability of "new hand" is to gather information. It can be done by searching, it can be done by asking. We try to follow existing practices and make things intuitive. But there will be some unavoidable irregularity. Simple rules like: you _must_ create test file with the same name as your sorce file do not work. Let me say that I see too many project where various "mandatory" files are empty or contain only meaningless boilerplate. Since sometimes there are good reasons for irregularity I do not insist on rigid structure. But I would like to avoid irregularities which are just due to slightly different preferences. > > What is important is that test should be easy to write > > and easy to run. > > Agreed and I think it is important to be able to run only tests relevant > to my code. So I can run tests frequently during development not just > when submitting a potential commit. You can run a single test file. To avoid creating a lot of tiny files I put algebraic topology tests in bugs2016.input. Normally it has no unexpected failures -- if after your modification there are unexpected failures it is likely due to your change. And it tells you which test failed. Note that if there are more test for algebraic topology we can move them to separate file. > > But if your tests > > require separate action to run, then they will > > be ignored. > > Why not call these tests from the test framework? Well, first I needed extra tests since the ones you wrote had limited coverage. And I found some bugs which needed fixing so I needed a few more tests. For me during debugging tests in .input were much more convenient than test in .spad file: I could ')read' the input file to run all tests or cut and paste lines to run just a single test. At the end I decided that the tests in input file have enough coverage for homology. Extra reason is that your tests in Spad file have no convenient way to report results to testing framework: you return true and just print error messages. Of course, I could try to modify your tests to behave in expected way. But it was _much_ easier to add tests to 'bugs2016.input'. -- Waldek Hebisch -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
Re: [fricas-devel] Regression testing
On 25/09/16 23:31, Waldek Hebisch wrote: Concerning 'randomSimplicialComplex': random testing has some advantages and drawbacks. Main drawback is that results change, so checking for correctness is harder. Normal random tests have advantage that they can generate large number of examples including many which human would not generate. Your 'randomSimplicialComplex' looses advantage of random testing because it just uses already prepared examples. Well I was planning to expand these tests. In addition to the basic examples from the factory functions I was planning to have random sums and products of these complexes. I am a bit concerned that the code has only really been tested with simple examples. I would like greater confidence that some structures will not break the code and that the structures scale up without error. Concerning organization of tests, I do not buy argument that "it must be in the same file" OK, how about having an additional directory like src/algebra where there is a test file corresponding to every (eventually) spad file in src/algebra. I think it would really help if the test code and source code were 1:1. What is in which file matters mostlty for Makefiles and grep searches. Putting things different than source code means a lot of false hits from grep, which is a significant distraction. Also Makefile may get unnecessarly complicated. When doing "quick and dirty" fixes people will go directly to relevant function -- if you add something 20 lines below it may be as easily missed as something in completely different file. I think it is important to design these things from the point of view of someone new to FriCAS. Imagine, if you can, that I was new to FriCAS, that I wanted to contribute to the code, that I was not familiar with the test framework and that I believed in test driven development. Would I know where to put the test code? Would I have to resort to searching the source to find existing tests? What is important is that test should be easy to write and easy to run. Agreed and I think it is important to be able to run only tests relevant to my code. So I can run tests frequently during development not just when submitting a potential commit. But if your tests require separate action to run, then they will be ignored. Why not call these tests from the test framework? Martin B -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
Re: [fricas-devel] Regression testing
> When doing "quick and dirty" fixes ... There must be a reason why the word "dirty" is in this phrase. ;-) Ralf -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
Re: [fricas-devel] Regression testing
Martin Baker wrote: > > On 24/09/16 22:33, Waldek Hebisch wrote: > > ATM I have omited part dealing with > > "randomSimplicialComplex". This function is fishy, giving > > result which is far from beein random. And usefulness is > > far from clear, for testing testing all cases is more > > efficient and more convenient. > > This function is for regression testing. I think it is very important > that regression tests are in the same source file as the code being > tested. It may seem that it is just as good to have a separate script > file like bugs2016 but I think human nature means things donâ t get used > and updated by anyone modifying the code if hidden away in a separate file. > > Of course you may have other regression tests but I think its also > important to have tests written by the original author and anyone else > working on the file. > > These tests have already caught a couple of bugs which might otherwise > had got into the code. > > It would be good if there was some standardised way of doing this that > does not pollute the namespace or use runtime resources. I do feel > strongly that this code does need to be in the same file or they could > be ignored. > > I would also like to write more regression tests for other code that I > have written. If there were a standarised way to add it to the source > files then I would do this. There are two things here: - 'randomSimplicialComplex' - organization of tests Concerning 'randomSimplicialComplex': random testing has some advantages and drawbacks. Main drawback is that results change, so checking for correctness is harder. Normal random tests have advantage that they can generate large number of examples including many which human would not generate. Your 'randomSimplicialComplex' looses advantage of random testing because it just uses already prepared examples. Concerning organization of tests, I do not buy argument that "it must be in the same file". What is in which file matters mostlty for Makefiles and grep searches. Putting things different than source code means a lot of false hits from grep, which is a significant distraction. Also Makefile may get unnecessarly complicated. When doing "quick and dirty" fixes people will go directly to relevant function -- if you add something 20 lines below it may be as easily missed as something in completely different file. What is important is that test should be easy to write and easy to run. Current test framework is far from ideal but it has provisions for collecting statistics and printing results. AFAICS you used 2-3 times as much code for a test than .input file using the framework. And 'make all-input' runs all tests. Anybody commiting to FriCAS is supposed to run tests and check results. But if your tests require separate action to run, then they will be ignored. -- Waldek Hebisch -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
[fricas-devel] Regression testing
On 24/09/16 22:33, Waldek Hebisch wrote: Since in similar way one can build projective space I have replaced projectivePlane() by projectiveSpace(n) where n is the dimension. I also made a few other changes to make code more efficient. Thank you for improving and committing. > ATM I have omited part dealing with "randomSimplicialComplex". This function is fishy, giving result which is far from beein random. And usefulness is far from clear, for testing testing all cases is more efficient and more convenient. This function is for regression testing. I think it is very important that regression tests are in the same source file as the code being tested. It may seem that it is just as good to have a separate script file like bugs2016 but I think human nature means things don’t get used and updated by anyone modifying the code if hidden away in a separate file. Of course you may have other regression tests but I think its also important to have tests written by the original author and anyone else working on the file. These tests have already caught a couple of bugs which might otherwise had got into the code. It would be good if there was some standardised way of doing this that does not pollute the namespace or use runtime resources. I do feel strongly that this code does need to be in the same file or they could be ignored. I would also like to write more regression tests for other code that I have written. If there were a standarised way to add it to the source files then I would do this. Martin B -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To post to this group, send email to fricas-devel@googlegroups.com. Visit this group at https://groups.google.com/group/fricas-devel. For more options, visit https://groups.google.com/d/optout.
