Joshua Herman wrote:
Is there a mathematica test suite we could adapt or a standardized set
of tests we could use? Maybe we could take the 100 most often used
functions and make a test suite?
I'm not aware of one. A Google found very little of any real use.
I'm sure Wolfram Research have such test suites internally, but they are not
public. There is discussion of how they have an internal version of Mathematica
which runs very slowly, but tests things in greater detail.
http://reference.wolfram.com/mathematica/tutorial/TestingAndVerification.html
Of course, comparing 100 things is useful, but comparing millions of them in the
way I propose would more likely show up problems.
I think we are all aware that it is best to test on the hardware you are using
to be as confident as possible that the results are right.
Of course, Wolfram Research could supply a test suite to check Mathematica on an
end user's computer, but they do not do that. They could even encrypt it, so
users did not know what was wrong, but could at least alert Wolfram Research.
I'm aware of one bug in Mathematica that only affected old/slower SPARC machines
if Solaris was updated to Solaris 10. I suspect it would have affected newer
machines too, had they been heavily loaded. (If I was sufficiently motivated, I
would probably prove that, but I'm not, so my hypothesis is unproven).
It did not produce incorrect results, but pegged the CPU at 100% forever if you
computed something as simple as 1+1.) It was amazing how that was solved between
myself, Casper Dik a kernel engineer at Sun and various other people on the
Internet. It was Casper who finally nailed the problem, after I posted the
output of lsof, he could see what Mathematica was doing.
I've got a collection of a few Mathematica bugs, mainly affecting only Solaris,
although one affected at least one Linux distribution too.
http://www.g8wrb.org/mathematica/
One thing I know Mathematica does do, which Sage could do, is to automatically
generate bug report if it finds a problem. At the most primitive level, that
code might be
if (x < 0)
function_less()
else if (x == 0)
function_equal()
else if (x > 0)
function_greater()
else
function_error()
If the error is generated, a URL is given, which you click and can send a bug
report to them. It lists the name of the file and line number which generated
the error. That's something that could be done in Sage and might catch some bugs.
Dave
---- LOOK ITS A SIGNATURE CLICK IF YOU DARE---
http://www.google.com/profiles/zitterbewegung
On Wed, Mar 3, 2010 at 12:04 AM, David Kirkby <david.kir...@onetel.net> wrote:
Has anyone ever considered randomised testing of Sage against Mathematica?
As long as the result is either
a) True or False
b) An integer
then comparison should be very easy. As a dead simple example,
1) Generate a large random number n.
2) Use is_prime(n) in Sage to determine if n is prime or composite.
3) Use PrimeQ[n] in Mathematica to see if n is prime or composite.
4) If Sage and Mathematica disagree, write it to a log file.
Something a bit more complex.
1) Generating random equation f(x) - something that one could integrate.
2) Generate generate random upper and lower limits, 'a' and 'b'
3) Perform a numerical integration of f(x) between between 'a' and 'b' in Sage
4) Perform a numerical integration of f(x) between between 'a' and 'b'
in Mathematica
5) Compare the outputs of the Sage and Mathematica
A floating point number, would be more difficult to compare, as one
would need to consider what is a reasonable level of difference.
Comparing symbolic results directly would be a much more difficult
task, and probably impossible without a huge effort, since you can
often write an equation in several different ways which are equal, but
a computer program could not easily be programmed to determine if they
are equal.
One could potentially let a computer crunch away all the time, looking
for differences. Then when they are found, a human would had to
investigate why the difference occurs.
One could then add a trac item for "Mathematica bugs" There was once a
push for a public list of Mathematica bugs. I got involved a bit with
that, but it died a death and I became more interested in Sage.
Some of you may know of Vladimir Bondarenko, who is a strange
character who regularly used to publish Mathematica and Maple bugs he
had found. In some discussions I've had with him, he was of the
opinion that Wolfram Research took bug reports more seriously than
Maplesoft. I've never worked out what technique he uses, but I believe
is doing some randomised testing, though it is more sophisticated that
what I'm suggesting above.
There must be a big range of problem types where this is practical -
and a much larger range where it is not.
You could at the same also compare the time taken to execute the
operation to find areas where Sage is much faster or slower than
Mathematica.
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
