On 08/29/10 07:07 AM, Robert Bradshaw wrote:
On Fri, Aug 27, 2010 at 12:37 PM, Dr. David Kirkby
There have been many reports of where doctests fail when running
make ptest
or
make ptestlong
which later pass if run individually. I've lost count of the number of times
I've seen that reported, on every operating system under the Sun.
I'm willing to bet this is a problem with the doctesting system, not
Sage itself. (Actually, there's at least one known bug that I've
personally run into when running at high levels of parallelization:
http://trac.sagemath.org/sage_trac/ticket/9739 )
I tend to agree.
I've run the parallel doctests 100 times now, and are now in the process of
running the serial doctest 100 times using the same compile of Sage. I will see
if serial is more reliable. I'm testing in 12 different directories, to speed
the process up. Otherwise it would take too long to run the long tests serially
100 times.
I ran the eigenvalues test 100,000 times on my Mac (Sage 4.5.2) and
never had it fail once. It looks like linbox is called to create the
charpoly, and IIRC linbox occasionally uses probabilistic methods by
default (with tiny but non-zero chance of failure). I wouldn't rule
out cosmic rays or other hardware failure--is your ram ECC? With 12GB,
you're likely to see a bit flipped per day.
Yes, the machine has ECC RAM. It's also not a cheap PC that's been overclocked.
For example the RAM is 1333 MHz, but Sun reduce the clock speed of the RAM when
there's more than 6 GB. So it runs at a bit under that (I forget what). The
disks are enterprise grade disks, mirrored, with a ZFS file system which should
detect problems, correct them and record them. I've never had a single disk
problem. (The machine is under a year old anyway).
Of course it could be a hardware error, but if so it was not logged as such. But
I've only seen that error once, and can't reproduce it, unlike some other
issues, which seem to be related to pexpect. But I'll post more details later,
when I have done some more testing.
Also, you're using a
(relatively) uncommon operating system and set of hardware.
Yes.
This can
be good for exposing bugs, but that's not a good thing when you're
trying to use the software. I have more confidence in software working
correctly on the system(s) used by its developers than a port.
Yes, I can see that. Though in mitigation I'd say that in the case of Solaris
(but not OpenSolaris), the underlying operating system is probably more stable
than a typical (if not all) Linux systems. But I am using OpenSolaris on this
machine, despite I know its not as stable as Solaris.
In terms of the general rant, there are two points I'd like to make.
The first is that there's a distinction between the Sage library
itself and the many other spkgs we ship. By far the majority of your
complaints have been about various arcane spgks. Sage is a
distribution, and we do try to keep quality up, but it's important to
note that much of this software is not as directly under our control,
Robert, I was going to reply earlier before I see Tim's post, but basically I
was going to say exactly the same has him. There is simply no logic to this
argument.
Sage is a collection of bash/python/perl/C/Fortan/Lisp etc and part of that
collection is the Sage library. But that is Sage. I think I used the analogy the
other day, Airbus can't say when a plane crashed "its not our fault, we did not
make the part that failed". Airbus are responsible for the design of the
aircraft. What they chose is up to them. It's a design decision. Sage made a
design decision to include all these .spkg files.
I think Tim has made this point, so I don't really need to say more.
and just because something isn't as good as it could be from a
software engineering perspective doesn't mean that it won't be
extremely useful to many people.Even if it has bugs.
Agreed. But from my own perspective, as a non-mathematician, I don't fell I can
trust Sage enough to do work that I'd want to do with it, because I have serious
concerns about the development process.
We try to place
the bar high for getting an spkg in but blaming the Sage community for
poor coding practices in external code is a bit unfair.
As Tim and I both point out, that is not a valid argument.
I hold the
Sage library itself to a much higher standard.
I've not looked at that in detail, but I see comments from people developing
code in the library that leads me to think I would not trust too much what they do.
The second point is that much of the older, crufty code in Sage was
put in at a time when standards were much lower, or even before there
was a referee process at all. I think this was necessary for the
time--Sage would have gotten off the ground if it couldn't have been
useful so quickly.
I have some sympathy with that view, though not an awful lot.
This includes in particular many of the spkgs that
have been grandfathered in and wouldn't make the cut now, but it takes
time to remove/replace/clean them up. Of course there's room for
improvement, but do you think the current review process is
insufficient and lots of new bad code is being written and included?
I don't want to get into a rant, but there's one developer, who I would rather
refer to anonymously, but he complained when I did that, so I'll name him -
Nathann Cohen.
He seems a nice guy, but wrote in an email that was circulated to about 6 people
(myself and William included), that he did not want to worry too much about how
code might fail, but rather fix the problems if there are bugs reported. I think
that is very bad practice. But not one single person on that list pointed out
this flaw in this logic. I raised the issue - perhaps I overreacted, but there
was nobody that actually told him that his methods were wrong.
I think there's simply a lack of the right mix of skills in developers.
If so, what should we do better?
* I think a good start would be to try to attract some compute science students
to do some projects looking at how quality could be improved. In essence, I
don't believe Sage has the right mix of skill sets. An interesting project or
two could be made by getting someone with a more appropriate skill set to come
up with some suggestions.
Doing this, might broaden the user base of Sage too.
* I don't know if William has funding to buy, and encourage developers to read
some books on software engineering. I think there's a lack of awareness of what
is considered good practice, and what is just asking for trouble.
Software engineering is not my background, though I believe I know more about it
than many of the developers. That's only because I've realised that to be
professional at something, one must know what the professionals do. As a
chartered engineer, I try to maintain professional standards.
He can buy me a copy of
http://www.amazon.com/Software-Engineering-9th-Ian-Sommerville/dp/0137035152/ref=sr_1_1?ie=UTF8&s=books&qid=1283077786&sr=8-1
if he wants.
* If there was funding, then pay someone with a strong background in a
commercial environment with knowledge of how to develop software. Someone good
would cost a lot of money. He/she should be interviewed by someone who knows the
subject well. Perhaps ask a prof from the CS dependent to be on an interview panel.
* Have regular "bug fix only" releases, where the aim is just to increase the
quality, and not to add features.
Nathann Cohen has said "-1" to that idea. William has said it would put people
off, and at least one other developer (forget who) did not like it. But I feel
it's probably one of the easiest ways to improve Sage.
* Have a system like gcc, where the releases numbers x.y.z mean something. Only
have z != 0 on bug-fix-only releases. Make it clear on the web site that the the
x.y.0 are less well tested.
* Make release candidates available for anyone to report on.
* Have longer times between the "final" release candidate and a release date. I
expressed concern that 4.5.3.alpha2 was going to be released last Monday and
4.5.3 released on Friday. Luckily that has not happened.
* Something I suggested before, which would not be perfect, but would be useful,
is to have a "risk factor" on trac tickets.
* I think William is spot on this blog post.
http://389a.blogspot.com/2010/08/computational-projects-part-1.html
There should be a difference in code quality that one developers for oneself,
and what gets put into Sage. It sounds like William will be developing things
for himself, which wont be committed to Sage, as he will not have time to
document/test them sufficiently well. That's a good sign.
But I think a lot of the code in Sage is very specialised things, that are only
useful to a very small number of people. IMHO, that should be in external
packages which people include if they want them. These bits of code will be
unmaintainable if the person including them leaves. I don't think they really
should be in the core Sage code. I think the R model is more appropriate.
* Spend more time on implementing well the things we have, rather than go onto
something else. There was for example a complaint on sage-support about a lack
of documentation for plotting 3D, with not all options documented.
Someone said the R interface is "rough around the edges". I'd like to see less
emphasis on sticking things into Sage, and a bit more on not having things that
are "round around the edges".
* Have design documents, which document how specific areas will be done. It
seems at the minute that someone has an idea for new bit of code, they create
some code for the library, it gets reviewed and committed. I don't see any real
design documents.
* Run the self-tests for the packages. In many caes upstream packages have
test code, but the people that added the .spkg's never bothered looking at
running the tests.
* Tim, who clearly knows more about maths than me, has also given a list.
* There are more I could think of, but its pointless listing them all. It
really does need someone with knowledge of best industry practices to look at
what happens.
* Stable and unstable branches, though that might be impractical due to the
lack of people wanting to be release managers. I think the bug-fix-only releases
would reduce the need for this a little, especially if people wanting a stable
release were advised to download a bug-fix-only release.
- Robert
BTW, there's an useful little post on the Wolfram Research web site, where
people make comments about the next Mathematica release
http://blog.wolfram.com/2009/11/12/the-rd-pipeline-for-mathematica/
I guess the comments are genuine, but of course they are moderated. But a couple
strike me as showing what *users* of software want when they get a package like
Mathematica, and I think will apply to many end users (non-developers) of Sage.
==========================================================================
I’d think the timescale involved is a couple of years, not months.
Let WRI use that time to get it right. Any releases with such amazing features
should be industrial-strength, not proof-of-concept. (Please, no more
proof-of-concept.)
Posted by Vince Virgilio
===========================================================================
I second Vince. Take the time to make sure that when it is released it is as
near to bug-free as possible.
Posted by Paul
============================================================================
Sorry the post is so long.
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
