On Sun, Apr 20, 2008 at 11:13 AM, TimDaly <[EMAIL PROTECTED]> wrote: > > > > How does the fricas/axiom source code layout work? > > Is it all written in pamphlets that lisp is generated from? > > There is a bit of a philosophical split between Axiom and > Fricas about source code layout and it is fairly fundamental.
Thanks for your explanation of this, which I think was very helpful. > Axiom has everything in pamphlet files and is gradually moving > to book-style layout. (As far as I know this is the only project > that is structured this way). One of Axiom's fundamental goals > is to restructure the complete source tree to be literate > programs. See the Lisp in Small Pieces or Tex, The Program books. > > Fricas is a project fork and one of its fundamental goals is to > remove the literate programming structure. This is an ongoing > effort (as evidenced by the post in late March, 2008: > <http://groups.google.com/group/fricas-commit/browse_thread/thread/ > ba4f3be7d257fef?hl=en> > So I really cannot comment on the non-Axiom processes. > > In Axiom all of the source code lives in Knuth-style literate > documents. So the basic process is to extract the source code > from the document and then compile it. This can involve several > steps since the algebra code has its own language (Spad). Thus > algebra goes thru the steps: > > extract spad from pamphlet -> > compile code with spad compiler to lisp -> > compile lisp code with GCL to C -> > compile C code to machine code with GCC What kind of runtime/build dependencies does the code generated by GCL have? It would be cool if Axiom/Maxima/whatever could be built as a pure C program, with no lisp involved at all, and GCL were only used say on linux to do the lisp --> C conversion. I have no idea if this makes any sense, but this is what happens with Cython. > > Anyway, I would love if somebody who knows what they > > are talking about regarding axiom (not me!) would > > explain what the human-written/readable code > > parts of the axiom distro are > > In the near term every "source code file" is a literate document. > The idea is that you should be able to open (using a div/pdf > viewer) the file and read it like a book. > > The rational for this huge change is based on experience. I > was one of the original authors of Axiom. I got my own code > back after 15 years and found it unreadable. This is despite > my best efforts at the time to write dirt-simple, clear code. > (You will encounter the same problem with Sage in the future.) Some people are a lot better at jumping into random code and making sense of it than others. My thesis adviser -- Hendrik Lenstra -- didn't code, but he was amazingly good at jumping into "random mathematics" and making sense of it. Reading code is similar. Actually, this whole problem you'll describing is a problem also in mathematics research. When a person decides to read a serious research paper in mathematics, they will often allocate weeks (at least) to really understand the paper (especially if they can't talk to any experts in the area). I'm not talking about computational math here, but just research mathematics in general, which is really the culture I was trained in. > I spent some time reading and pondering this issue and eventually > discovered that Knuth had already solved it using Web technology > for literate programs. I STRONGLY encourage you to learn about > this technology because Sage is going to have the same fundamental > problems that Axiom encountered. I'm unconvinced that literate programming is a silver bullet that solves the problem of making code easier to understand later. I do strongly advocate documenting and writing excellent test suites for code. All code that goes into Sage has every function tested, documented, and it is refereed by a different person (who may as well be somebody reading the code 15 years later...). I think peer review is a valuable step in the right direction toward solving the problem you are addressing. That's what's done in mathematics to increase the chances that somebody can understand a math paper 15 years after it is written. > One problem is that the research is not associated with the source > code. A recent example is the Pollard-Rho algorithm that was part > of the Sage/Fricas discussion. It turns out that Axiom implements > Brent's algorithm which is an extension of Pollard-Rho, despite > the comments to this list. The only way to know that is to read > the code, read the literature, and "discover" what the code does. > > Sage is being built by experts who are the primary source for the > material. This is excellent while it lasts but it won't last. Why won't it last? I mean, the experts will change, but I see no reason that the statement "Sage is being built by experts ..." will not last. > At some point in the future these experts will no longer be available. I guess that's true. At some point in the future nobody in particular will be available. > If you think long term it becomes clear what this implies. How longterm are you thinking? > Code is guaranteed to be opaque because the world's expert in some > area (L-functions?) wrote clever, highly optimized code that does > not correspond to anything in the literature. Thus, the only person > who can properly maintain, modify, and extend the code will no > longer be available and the code will be frozen in time. I simply do *NOT* have such a pessimistic view of the capabilities of the people who work on Sage. > My Magnus infinite group theory project has this problem. > > New people coming onto the project cannot find the literature > that corresponds to the actual code because it does not exist. > Most published results are 5 page conference papers that just > hint at the non-core, but vitally important, details if they > mention them at all. This problem has 2 parts, both of which > stem from the lack of focus on the new discipline of computational > mathematics. Part 1 is that journals and conferences only want > short papers, which are adequate for math, not the complete > implementation. Part 2 is that there is no place to describe the > actual code-level optimizations that make the implementation > practical and fast. I assume your talking specifically about your project(s) and not all computational mathematics projects. Since the above is not true of all such projects. > I believe that the Knuth's literate programming technology will > solve the long term problem and keep the code as a living, > understandable, maintainable, and extensible object. It directly > addresses the problem of bringing the research and the code together. > It gives a place to explain both the algorithm and the current > code optimizations. Further, I've been working with Carlo Traverso > (Univ. of Pisa, Math) to create a journal of these literate > programs so the algorithms and their implementations can be made > widely available for study and improvement. > > This is a controversial position and led to a philosophical > split between Axiom and Fricas. Fricas has the stated goal of > competing with Mathematica. Axiom plans to change the game so > Mathematica is irrelevant. I'm surprised by how convinced you are that using a specific technology/language -- literate programming -- can be a silver bullet to solve such a difficult problem. I think peer review, and many many other things, are steps in the right direction, but *not* solutions to the problem. I think it's just a difficult problem, with no easy solutions. > > and roughly how big each is, in some sense. > > Ah, size. Here is where Axiom parts company with most of the > other systems. Axiom is highly structured using categories > and domains. It is trivially easy to construct a domain which > overrides the categorical definition of a single function > but uses all of the rest by inheritance. I think Magma (and Sage) are exactly like that. > Thus, "line of code" have nothing to do with "conceptual size". > > That said, Axiom is still huge by any measure. It represents > about 30 years and 300 man-years of research. It covers a > wide range of computational mathematics. Sorry I was mainly interested in how many lines of code, in some rough sense, are in the Axiom codebase, since this whole thread started discussing the difficulties involved with the code size of Axiom (since it's about combining source code for project). I'm still really curious how big the Axiom codebase is. > > > > > Or just point me to an article or wiki page > > about this. And who are some of the Axiom original > > authors? Some files have headers like: > > > > ++ Author: Grabmeier, Gschnitzer, Williamson > > ++ Date Created: 1987 > > ++ Date Last Updated: July 1990 > > > > I wonder who those guys were...? > > Ummm, who these guy *ARE*. > > There is a very active community based around the ISSAC > (International Symposium on Scientific and Algebraic > Computation) and ECCAD (East Coast Computer Algebra Days) > which is attended by almost everyone in the computational > mathematics community. It has a long history. There are > a large number of people still active from the early Axiom > days. You can see a list of the some of the people by > starting Axiom and typing )credits. Sage may be new but > we've been at this for a long time :-) So have I. I've been seriously involved in writing mathematical software for over 10 years. I worked as a developer on Magma for 5 years... > Sage should definitely concentrate on presenting at ISSAC. > You could probably start a Sage paper track. Since I know > most of the people involved I'd be willing to help. I'm speaking at the next ISSAC... we'll see how it goes. If that community likes Sage, then I bet a lot of positive things will come of it. I really don't know what to expect. > > There are rumblings but *definitely* no specific plans to remove > > lisp or maxima. > > Lisp has already solved a lot of the problems Python has yet to > face. But since that discussion is a prelude to a language war > I'll end it here. That's where you're a *lot* more fun to discuss mathematical software with that Richard Fateman! Thanks for taking the time to write. > > > > > > That might sound weird, since why would an Axiom guy want to > > help with working on or improving Maxima? > > While both are coded in Lisp they are philosophically wildly > different. Axiom algebra is in the spad language, not Lisp. > Thus "fixing maxima" (which works fine, btw) would be like > working in a completely separate language. I know James > (the Maxima Lead) and we're not in any kind of competition. I have the impression that Waldek Habisch is extremely talented at hacking build systems and building code that involves lisp. I wouldn't be surprised if he could make many positive contributions toward getting Maxima to run on http://ecls.sourceforge.net/. > > > But that's the > > ** Sage WAY ** which is to get all open source software packages > > to work better. I think any and all competition/fighting between > > open source math software is completely counterproductive. > > The goal we should all have is to provide a viable free open > > source alternative to Maple, Mathematica, Matlab, and Magma. > > While I agree with "free" and "open source" (given that I've spent > the last 7 years doing that with Axiom) I have to question the > "viable" point. I think "viable" needs to be discussed within > Sage because it is vital. What do you mean? Do you mean that disagree with making mathematical software that is viable? Instead you only want to make mathematical software that is free and open source, but isn't viable? > > If you looked at the list of credits from Axiom mentioned above > you'll notice that, of the 140+ contributors, some of them are dead. > I just attended a funeral for one in January. Sage also unfortunately has a deceased contributor listed at http://lite.sagemath.org/devmap.html > Sage is too new to think about this problem but what happens when > your expertise "disappears"? It happens all the time. Iregardless of whether developers die or not, they do disappear -- e.g., they go into industry instead of academia and are legally obliged to no longer contribute (or just not interested) -- or they move onto other things -- or get a job with an incredibly high teaching load. It happens all the time. What happens? You deal with it. You realize that it is going to keep happening, so you constantly build in a wide range of strategies to protect against it. This is the way it is with all large software projects, or indeed all large group enterprises. > What happens when you are the only > use of an open source program (such as Magnus, an infinite group > theory package)? It typically dies. > Suppose the maintainer quits or dies? If there is only one user and that person quits, who cares? > What will > Sage do (WWSD)? > Do you just drop the package? Do you let it rot? We don't like having things in Sage with only 1 users in the entire world. In fact, if there is only 1 and they vanish, who cares? > Will I find that my notebooks won't work in the next release? > If Magnus fails to compile under C++0X do you just drop it? We don't include Magnus in Sage. > Do you try to recode it in Python? Do you know enough infinite > group theory to recode it? Do you know anyone willing to recode > it? Do you even know what algorithms it has? No. No. No. No. We would ask all those questions if somebody proposed adding Magnus to Sage. All those no answers would likely cause people to vote against it and it wouldn't get included. > I would hope to convince the Sage team that THE most important > contribution they can make is to provide a firm base for future > computational mathematics. Think of the science and the long > term. Insist that people document their code in a literate form > so that others can maintain and extend the code. We insist that people document their code and that it passes peer review. > Your REAL issue is not the "4Ms". Yes it is. It's *my* personal real issue at least. > They are going to die because all > companies die and they are going to disappear because dying > companies take their code with them (witness Symbolics and > Macsyma). All that matters to me regarding the above comments are the *next* 30 years, since after that I'll likely be "done". And it's possible that none of Maple/Mathematica/Matlab will die in the next 30 years. > The REAL issue is the science of computational > mathematics. Not for me. I just want to "finish" Sage as soon as possible, so I can do research and teaching using it. From this point of view, Sage is surprisingly close to where I want. It's I think at most 1 or 2 years away. One important thing that we aren't yet close on is very good native Microsoft Windows support; and I don't mean crippled Cygwin support or virtual machines. > Open source suffers from "project rot". Lead developers and > maintainers move on to other things and the projects die. > (Witness "Pinger", which was my open source implementation of > a closed source network management tool for Worldcom, another > company that "couldn't disappear" but did.) Some open source projects don't die. Witness Linux, which is an operating system that "could easily disappear" but didn't. > > because Maxima (via pexpect) is too slow and, > > much more importantly, it is too hard for developers to improve > > and change. This difficulty for developers has stopped dead > > promising Sage development projects related to symbolic > > calculus computation. This symbolic manipulation rewrite > > is actually well on its way; Gary Furnish has been working on > > it day and night for a few months now, and has been funded > > by the NSF and Google (thanks!) to work on it full time at UW > > all summer. > > Gee, wouldn't it be great if Gary could just read the literate > documentation of a system like Maxima and know the details of > how to implement the algorithm? > This IS the late 90s and we can ^^^^^^^^^^^^^^^^^^^^ Last I checked it is not the late 90s. ??? > construct real documentation of real computational mathematics > if we want so that the science as a whole benefits. I hope that > Gary generates literate documents so I can re-implement his > algorithms in Axiom. I'm sure he's not generating literate documents. He's probably also mostly not coming up with any new algorithms easier. He's just doing what he is doing because he is a very talented programming and he needs the results to support his research interests. -- William _______________________________________________ Axiom-developer mailing list Axiom-developer@nongnu.org http://lists.nongnu.org/mailman/listinfo/axiom-developer