Re: [sage-devel] Re: Random testing for finite lattices
On Wed, 30 Aug 2017, Travis Scrimshaw wrote: What if we add tests of, say, random graphs to this file? Or should we have basically one function in one file? It doesn't have to necessarily be one function, but I think it would be good to keep it closer to one specific type of object (at least, I think of posets as being very different from (random) graphs). OK. I made this as finite_poset.py, so we can add other poset code here too. Later we could have graph.py for both undirected and directed graphs and so on. I should have said _test_* method, but any method that starts like that will be run by the TestSuite. See categories/sets_cat.py or categories/semigroups.py for example. That seems to a different thing. What I suggest is mostly cross-checking over functions: if L.is_distributive() != L.dual().is_distributive(), then either is_distributive() is broken or dual() is broken, but we don't know which one. -- Jori Mäntysalo
Re: [sage-devel] Re: Random testing for finite lattices
On Wednesday, August 30, 2017 at 12:27:04 PM UTC-5, Jori Mäntysalo wrote: > > On Wed, 30 Aug 2017, Travis Scrimshaw wrote: > > > Looks good (other than call the file something like "lattice_tests.py"). > > What if we add tests of, say, random graphs to this file? Or should we > have basically one function in one file? > It doesn't have to necessarily be one function, but I think it would be good to keep it closer to one specific type of object (at least, I think of posets as being very different from (random) graphs). > > > Now to fully flush it out. Once all the tests are in there, we can see > > how many of them are better lifted to the category of > > FiniteLattticePosets as _tests_* methods. > > I think I didn't understood. Categories of posets ja lattices contains > just few functions. A PoC code? > > I should have said _test_* method, but any method that starts like that will be run by the TestSuite. See categories/sets_cat.py or categories/semigroups.py for example. Best, Travis -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: Random testing for finite lattices
On Wed, 30 Aug 2017, Travis Scrimshaw wrote: Looks good (other than call the file something like "lattice_tests.py"). What if we add tests of, say, random graphs to this file? Or should we have basically one function in one file? Now to fully flush it out. Once all the tests are in there, we can see how many of them are better lifted to the category of FiniteLattticePosets as _tests_* methods. I think I didn't understood. Categories of posets ja lattices contains just few functions. A PoC code? -- Jori Mäntysalo
Re: [sage-devel] Re: Random testing for finite lattices
> > So the use would be something like
> >
> > sage: from some.place.tests import test_finite_lattice
> > sage: for L in big_list_of_random_lattices:
> > sage: test_finite_lattice(L)
> > sage: print("All OK")
>
> > Yes, but I would not do the print().
>
> I made https://trac.sagemath.org/ticket/23754
>
>
> to see if I understood this
> right.
>
> Looks good (other than call the file something like "lattice_tests.py").
Now to fully flush it out. Once all the tests are in there, we can see how
many of them are better lifted to the category of FiniteLattticePosets as
_tests_* methods.
> > > This could also be used as a function to run timing benchmarks
> > on Sage
> > > as well.
> >
> > Not really, as some tests take so much more time. But they could
> > find a
> > bug-like time regression in some specific function
> >
> > It would have to be done manually (at least at present, there are some
> > people who are [strongly?] interested in having a benchmark framework
> for
> > Sage). However, it still could help.
>
> For a Posets.BooleanLattice(6) it takes 3 milliseconds to see if the
> lattice is modular, and maybe 2 seconds to see if it is isoform. So, if
> running time of is_modular() raises by factor of 10, it would just mean
> that the combined time would raise from 2,003 seconds to 2,030 seconds.
> Hence this is only useful for a) very pathological regression or b)
> smaller regression that happens in some slow function.
>
> For the faster functions, it is less likely to catch regressions, but it
is still something we could measure. However, speed regression testing is
not its primary purpose, but it is a nice side effect of stress testing.
Best,
Travis
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: Random testing for finite lattices
On Wed, 30 Aug 2017, Travis Scrimshaw wrote:
So the use would be something like
sage: from some.place.tests import test_finite_lattice
sage: for L in big_list_of_random_lattices:
sage: test_finite_lattice(L)
sage: print("All OK")
Yes, but I would not do the print().
I made https://trac.sagemath.org/ticket/23754 to see if I understood this
right.
> This could also be used as a function to run timing benchmarks
on Sage
> as well.
Not really, as some tests take so much more time. But they could
find a
bug-like time regression in some specific function
It would have to be done manually (at least at present, there are some
people who are [strongly?] interested in having a benchmark framework for
Sage). However, it still could help.
For a Posets.BooleanLattice(6) it takes 3 milliseconds to see if the
lattice is modular, and maybe 2 seconds to see if it is isoform. So, if
running time of is_modular() raises by factor of 10, it would just mean
that the combined time would raise from 2,003 seconds to 2,030 seconds.
Hence this is only useful for a) very pathological regression or b)
smaller regression that happens in some slow function.
--
Jori Mäntysalo
Re: [sage-devel] Re: Random testing for finite lattices
On Wednesday, August 30, 2017 at 1:16:37 AM UTC-5, Jori Mäntysalo wrote:
>
> On Wed, 23 Aug 2017, Travis Scrimshaw wrote:
>
> >> Thank you for doing this. Do you want to add this as a file into the
> >> $SAGE_SRC/sage/tests? How long does it take to run these tests on your
> >> system?
>
> > Since they are around 1 second, you could add the test code to our tests
> folder with one to a few "generic" examples
> > marked with "# long time". That way anyone who wants to run more stress
> testing can easily do so (as opposed to, e.g.,
> > posting the code on your homepage).
>
> So the use would be something like
>
> sage: from some.place.tests import test_finite_lattice
> sage: for L in big_list_of_random_lattices:
> sage: test_finite_lattice(L)
> sage: print("All OK")
>
> ?
>
Yes, but I would not do the print().
>
> > This could also be used as a function to run timing benchmarks on Sage
> > as well.
>
> Not really, as some tests take so much more time. But they could find a
> bug-like time regression in some specific function -- if something very
> odd happens and for example is_modular() takes a minute instead of few
> milliseconds, it would be noticed.
>
> It would have to be done manually (at least at present, there are some
people who are [strongly?] interested in having a benchmark framework for
Sage). However, it still could help.
Best,
Travis
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: Random testing for finite lattices
On Wed, 23 Aug 2017, Travis Scrimshaw wrote:
Thank you for doing this. Do you want to add this as a file into the
$SAGE_SRC/sage/tests? How long does it take to run these tests on your
system?
Since they are around 1 second, you could add the test code to our tests folder with one
to a few "generic" examples
marked with "# long time". That way anyone who wants to run more stress testing
can easily do so (as opposed to, e.g.,
posting the code on your homepage).
So the use would be something like
sage: from some.place.tests import test_finite_lattice
sage: for L in big_list_of_random_lattices:
sage: test_finite_lattice(L)
sage: print("All OK")
?
This could also be used as a function to run timing benchmarks on Sage
as well.
Not really, as some tests take so much more time. But they could find a
bug-like time regression in some specific function -- if something very
odd happens and for example is_modular() takes a minute instead of few
milliseconds, it would be noticed.
--
Jori Mäntysalo
Re: [sage-devel] Re: Random testing for finite lattices
On Wednesday, August 23, 2017 at 12:51:54 PM UTC-5, Jori Mäntysalo wrote: > > On Wed, 23 Aug 2017, Travis Scrimshaw wrote: > > > Thank you for doing this. Do you want to add this as a file into the > > $SAGE_SRC/sage/tests? How long does it take to run these tests on your > > system? > > It seems to already be more than a second per lattice. > > This kind of randomized stress testing can not be added to any normal > release cycle. I do not know if our current framework has any good place > for this. > Since they are around 1 second, you could add the test code to our tests folder with one to a few "generic" examples marked with "# long time". That way anyone who wants to run more stress testing can easily do so (as opposed to, e.g., posting the code on your homepage). This could also be used as a function to run timing benchmarks on Sage as well. > > Is there more places for this kind of testing? Personally I have run some > tests for QQ-polynomial factoring and found several bugs in Singular (they > have been corrected). For matrices we know that product of eigenvalues > equals to determinant etc, product of invertible matrices is invertible > etc. What else could be done? > > I've come across something that is possibly ZZ-polynomial bug in Singular, but I ended up working around it by converting to the SymbolicRing. I'm not sure if I would be able to recover it, much less isolate what precisely the problem was as I was doing stuff in the ZZ['q'].fraction_field(). Although for these types of bugs, individual doctests added in the corresponding code (merged once the corresponding Singular upgrade was done). Ralf regularly does this with Pynac. Further tests that would be good are symbolic and calculus related. There are a fair body of tests, but maybe some more specific integrals or expression simplifications to do both regression and speed testing would be useful. However, I think Ralf and other people who care and work more in the symbolics might have a better thought or opinion on this. For those things with matrices, these are more like unitests, and so I think it would be good to add to the TestSuite methods by adding methods named _test_*. Thus, something like _test_eigenvalues would test that the sum is the trace and the product is the determinant, etc. Best, Travis -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: Random testing for finite lattices
On Wed, 23 Aug 2017, Travis Scrimshaw wrote: Thank you for doing this. Do you want to add this as a file into the $SAGE_SRC/sage/tests? How long does it take to run these tests on your system? It seems to already be more than a second per lattice. This kind of randomized stress testing can not be added to any normal release cycle. I do not know if our current framework has any good place for this. Is there more places for this kind of testing? Personally I have run some tests for QQ-polynomial factoring and found several bugs in Singular (they have been corrected). For matrices we know that product of eigenvalues equals to determinant etc, product of invertible matrices is invertible etc. What else could be done? -- Jori Mäntysalo
[sage-devel] Re: Random testing for finite lattices
Thank you for doing this. Do you want to add this as a file into the $SAGE_SRC/sage/tests? How long does it take to run these tests on your system? Best, Travis On Tuesday, August 22, 2017 at 11:27:28 PM UTC-5, Jori Mäntysalo wrote: > > I have been doing some quality check for finite lattice code. This is a > work in progress, but already can give some ideas for others, so I make it > public for few weeks: > > https://sage.sis.uta.fi/home/pub/153/ > > -- > Jori Mäntysalo > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
