[sage-devel] Re: All of sage as a python library
Being able to get Sage as a part of PyPI would be great! Taking into account how many of Sage spkgs are there, e.g. cython, scipy, networkx, cvxopt, this looks like the right way of factoring out components that are just packaged into Sage. At the moment just keeping apace with the latter components is taking a large chunk of developers time. To begin with, I imagine one can look into ways PyPI manages dependencies, etc. I understand there is a mechanism that allows for pinning of particular versions, etc... On Oct 27, 9:44 am, William Stein wrote: > Hi, > > When I started Sage I viewed it as a distribution of a bunch of math > software, and Python as just the interpreter language I happen to use > at the time. I didn't even know if using Python as the language would > last. However, it's also possible to think of Sage as a Python > library. > > Anyway, it has occurred to me (a few times, and again recently) that > it would be possible to make much of the Sage distribution, without > Python of course, into a Python library. What I mean is the > following. You would have a big Python library called "sagemath", > say, and inside of that would be a huge HG repository. In that > repository, one would check in the source code for many of the > standard Sage spkg's... e.g., GAP, Pari, etc. When you type > > python setup.py install > > then GAP, Pari, etc., would all get built, controlled by some Python > scripts, then installed as package_data in the sagemath directory of > /site-packages/. > > From a technical perspective, I don't see any reason why this couldn't > be made to work. HG can handle this much data, and "python setup.py > install" can do anything. It does lead to a very different way of > looking at Sage though, and it could help untangle things in > interesting ways. > > (1) Have a Python library called "sagecore", which is just the most > important standard spkg's (e.g., Singular, PARI, etc.), perhaps > eventually built *only* as shared object libraries (no standalone > interpreters). > > (2) Have a Python library which is the current Sage library (we > already have this), and which can be installed assuming sagecore is > installed. > > (3) Have other Python libraries (like > psage:http://code.google.com/p/purplesage/source/browse/), which depend on > (2). Maybe a lot of the "sage-combinat" code could also be moved to > such a library, so they can escape the "combinat patch queue" madness. > Maybe many other research groups in algebraic topology, differential > geometry, special functions, etc., will start developing such > libraries... on their own, and share them with the community (but > without having to deal directly with the sage project until they want > to). > > To emphasize (3), when people want to write a lot of mathematics code > in some area, e.g., differential geometry, they would just make a new > library that depends on Sage (the library in (2)). We do the work > needed to make it easy for people to write code outside of the Sage > library, which depends on Sage. Especially writing Cython code like > this can be difficult and confusing, and we don't explain it all in > any Sage documentation. It actually took me quite a while to figure > out how to do it today (with psage). > > The core Sage library (2) above would continue to have a higher and > higher level of code review, tough referee process etc. However, the > development models for (3) would be up to the authors of those > libraries. > > The above is already how the ecosystem with Python > (http://pypi.python.org/pypi), Perl (http://www.cpan.org/), R, etc., > work. Fortunately, Python has reasonably good support already for > this. > > I think without a shift in this direction, Sage is going to be very > frustrating for people writing research oriented code. > > Fortunately, it's possible to do everything I'm describing above > without disturbing the mainline Sage project itself, at least for > now. > > -- William > > -- > William Stein > Professor of Mathematics > University of Washingtonhttp://wstein.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 27 Okt., 05:54, Dima Pasechnik wrote: > Being able to get Sage as a part of PyPI would be great! > Taking into account how many of Sage spkgs are there, e.g. cython, > scipy, networkx, cvxopt, > this looks like the right way of factoring > out components that are just packaged into Sage. > At the moment just keeping apace with the latter > components is taking a large chunk of developers time. Well, if all of their spkg-installs would just do "cd src && python setup.py install"... Also, upgrading a Python package frequently requires changing other parts of Sage. Otherwise updating such an spkg wouldn't make much of a difference to including an updated Python package. -Leif -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 27 Okt., 08:26, Dan Drake wrote: > On Tue, 26 Oct 2010 at 06:44PM -0700, William Stein wrote: > > (1) Have a Python library called "sagecore", which is just the most > > important standard spkg's (e.g., Singular, PARI, etc.), perhaps > > eventually built *only* as shared object libraries (no standalone > > interpreters). > > What about those people who install Sage because it's an easy way to get > a running version of Gap, Pari, etc? I second that. As long as Gap and Singular are not both *fully* usable as a library, I couldn't use "Sage without interfaces" for my work. Cheers, Simon -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
Hi! On Oct 27, 3:44 am, William Stein wrote: > The above is already how the ecosystem with Python > (http://pypi.python.org/pypi), Perl (http://www.cpan.org/), R, etc., > work. Fortunately, Python has reasonably good support already for > this. I think that going into this direction is some awesome step. It makes my Python heart beat at this cold autumn morning. The Sage community has been an enthusiastic, well organized community, who developed great functionality. Using Sage as distribution has provided this process a uniform environment. Moving more to the standard Python ecosystem is just a very similar step as the Zope community has done with their repoze project, making Zope tech. available to standard WSGI apps: http://repoze.org/about.html Cheers, Michael -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
Just to clarify, are we talking about different namespaces from sagecore.rings import Integers from sagemain.modular.all import euler_phi from sagecombinat.combinat import choose_nk This seems a bit unwieldy. On the other hand, if Sage pulls everything into sage.* then how do I know which library I'm using. On a related note, can the sagecore documentation link to the main sage library documentation? Volker On Oct 27, 2:44 am, William Stein wrote: > Hi, > > When I started Sage I viewed it as a distribution of a bunch of math > software, and Python as just the interpreter language I happen to use > at the time. I didn't even know if using Python as the language would > last. However, it's also possible to think of Sage as a Python > library. > > Anyway, it has occurred to me (a few times, and again recently) that > it would be possible to make much of the Sage distribution, without > Python of course, into a Python library. What I mean is the > following. You would have a big Python library called "sagemath", > say, and inside of that would be a huge HG repository. In that > repository, one would check in the source code for many of the > standard Sage spkg's... e.g., GAP, Pari, etc. When you type > > python setup.py install > > then GAP, Pari, etc., would all get built, controlled by some Python > scripts, then installed as package_data in the sagemath directory of > /site-packages/. > > From a technical perspective, I don't see any reason why this couldn't > be made to work. HG can handle this much data, and "python setup.py > install" can do anything. It does lead to a very different way of > looking at Sage though, and it could help untangle things in > interesting ways. > > (1) Have a Python library called "sagecore", which is just the most > important standard spkg's (e.g., Singular, PARI, etc.), perhaps > eventually built *only* as shared object libraries (no standalone > interpreters). > > (2) Have a Python library which is the current Sage library (we > already have this), and which can be installed assuming sagecore is > installed. > > (3) Have other Python libraries (like > psage:http://code.google.com/p/purplesage/source/browse/), which depend on > (2). Maybe a lot of the "sage-combinat" code could also be moved to > such a library, so they can escape the "combinat patch queue" madness. > Maybe many other research groups in algebraic topology, differential > geometry, special functions, etc., will start developing such > libraries... on their own, and share them with the community (but > without having to deal directly with the sage project until they want > to). > > To emphasize (3), when people want to write a lot of mathematics code > in some area, e.g., differential geometry, they would just make a new > library that depends on Sage (the library in (2)). We do the work > needed to make it easy for people to write code outside of the Sage > library, which depends on Sage. Especially writing Cython code like > this can be difficult and confusing, and we don't explain it all in > any Sage documentation. It actually took me quite a while to figure > out how to do it today (with psage). > > The core Sage library (2) above would continue to have a higher and > higher level of code review, tough referee process etc. However, the > development models for (3) would be up to the authors of those > libraries. > > The above is already how the ecosystem with Python > (http://pypi.python.org/pypi), Perl (http://www.cpan.org/), R, etc., > work. Fortunately, Python has reasonably good support already for > this. > > I think without a shift in this direction, Sage is going to be very > frustrating for people writing research oriented code. > > Fortunately, it's possible to do everything I'm describing above > without disturbing the mainline Sage project itself, at least for > now. > > -- William > > -- > William Stein > Professor of Mathematics > University of Washingtonhttp://wstein.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On Oct 27, 1:05 pm, leif wrote: > On 27 Okt., 05:54, Dima Pasechnik wrote: > > > Being able to get Sage as a part of PyPI would be great! > > Taking into account how many of Sage spkgs are there, e.g. cython, > > scipy, networkx, cvxopt, > > this looks like the right way of factoring > > out components that are just packaged into Sage. > > At the moment just keeping apace with the latter > > components is taking a large chunk of developers time. > > Well, if all of their spkg-installs would just do > "cd src && python setup.py install"... > > Also, upgrading a Python package frequently requires changing other > parts of Sage. Sure, but this would at least remove the burden of actually upgrading the corresponding spkg. And this burden seems to be extreme. CVXOPT upgrade saga is a good example of what I mean; somehow after a year and hundreds of comments on the corresponding ticket Sage still distributes CVXOPT 0.9.8, not CVXOPT 1.1.3. And this given the fact that there is about 1, yes, one doctest where CVXOPT is required, and about one function in Sage that uses it (and this function isn't even affected by such an upgrade!). On the other hand, if an upgrade of FOO on PyPI broke PyPI-Sage, one could pin the right version of FOO to be used, as far as I know > > Otherwise updating such an spkg wouldn't make much of a difference to > including an updated Python package. > > -Leif -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
> > (1) Have a Python library called "sagecore", which is just the most > important standard spkg's (e.g., Singular, PARI, etc.), perhaps > eventually built *only* as shared object libraries (no standalone > interpreters). > Interesting! One challenge I see here, is that on the one hand, Python is pretty agnostic to the OS resp. system it lives on. This has both advantages and disadvantages. It's good for everything "pythonic", but there are quite essential parts of Sage that are (or rely on) C/C++ libraries. Building these is sensitive to the OS resp. system, and the "classic" setuptools/distutils are ill suited to handle this (except if such libraries are already provided as binaries ...). The situation gets even worse taking Cython into account, whenever Cython code relies on some external C (or C++) libraries (dependency/library version checking, or think of "#clib" and friends). But even "pure Python" code is problematic, as soon as some external C (or C++) libraries are needed (PIL e.g. "just" needs libtiff, libjpeg, ... I certainly do remember it to be troublesome). The SciPy/NumPy project is in a quite similar situation. David Cournapeau, the maintainer of NumPy, has written up some details of this challenge in a blog post last year: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ Don't get mislead by the name of the link, the outcome is a (still work in progress) build system he named "bento", see: http://cournape.wordpress.com/2010/10/10/bento-0-0-4-released/ I just can't believe David Cournapeau would go this way, if "python setup.py install" could do "anything", or if Python resp. its ecosysem (PyPI, ...) already had reasonably good support for the needs of the NumPy project. However, the idea of having "layers", explicitly a "sagecore" Python library, on top of which sits the "sagestandard" Python library, on top of which (or at its side) may sit some research "psage" Python library, is IMHO a valuable one, worth of pursuing further. Essentially, it's a matter of slicing the current rather monolithic Sage distribution into reasonable peaces (whatever "reasonable" might mean in this context ...). Cheers, Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On Oct 28, 4:23 am, "Georg S. Weber" wrote: > > (1) Have a Python library called "sagecore", which is just the most > > important standard spkg's (e.g., Singular, PARI, etc.), perhaps > > eventually built *only* as shared object libraries (no standalone [...] > > I just can't believe David Cournapeau would go this way, if "python > setup.py install" could do "anything", or if Python resp. its ecosysem > (PyPI, ...) already had reasonably good support for the needs of the > NumPy project. > the point that David Cournapeau makes is about distutils/setuputils getting too messy, too procedural, as opposed to declarative. His point about the need of a better packaging system is not about the lack of power in the present system, it's about lack of structure and abundance of general mess... (Python borrows from Haskell now and then, so if it would borrow from Haskell's package system, this won't be bad :-)) Dima > However, the idea of having "layers", explicitly a "sagecore" Python > library, on top of which sits the "sagestandard" Python library, on > top of which (or at its side) may sit some research "psage" Python > library, is IMHO a valuable one, worth of pursuing further. > Essentially, it's a matter of slicing the current rather monolithic > Sage distribution into reasonable peaces (whatever "reasonable" might > mean in this context ...). > > Cheers, > Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 1 Nov., 07:51, William Stein wrote: > This post is about: > > (1) Concern about distutils/setuptools/etc., is misplaced. > (2) Python3 and librarifying Sage. > > First, all this discussion about distutils/setuptools/david > cournapeau, etc., is actually mostly IRRELEVANT to making the core > Sage library into a standalone library. That's an argument (honestly). Distutils/Setuptools is broken (or weird), so we don't [have to] use it. Not very "pythonic" though, I guess... ;-) (but nobody should be forced to drive with broken wheels) > [...] > 2. A function in setup.py builds all the non-standard C/C++ libraries > that the core Sage library depends on, which is the following 24 > libraries: > > boost-cropped givaro libm4ri mpir ratpoints > cliquer gsl libpng ntl > eclib iml linbox pari singular > ecm lcalc mpfi polybori symmetrica > flint libfplll mpfr pynac zn_poly > [...] I'd prefer having plain text files rather than a pickled build_db. Adding (formal) dependency specifications to the spkgs (a file, say, spkg-deps in each spkg) would be a step forward, too, such that we can also *generate* the "real Makefile" spkg/standard/deps in the traditional build process. (In addition, a lot of what's currently performed in every spkg- install could be factored out.) -Leif > [...] Thus if we wanted a Python3 version > of the Sage library itself, if we had a library like I describe above, > this would only require a Python3 version of numpy and networkx, plus > the work of porting the Sage library itself. This doesn't sound so > far off, since there already is a Python3 version of numpy. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
> > This post is about: > > (1) Concern about distutils/setuptools/etc., is misplaced. > (2) Python3 and librarifying Sage. > > First, all this discussion about distutils/setuptools/david > cournapeau, etc., is actually mostly IRRELEVANT to making the core > Sage library into a standalone library. The way it would work is > this: > > 1. You type "python setup.py develop" (or possibly "python setup.py > install"). > > 2. A function in setup.py builds all the non-standard C/C++ libraries > that the core Sage library depends on, which is the following 24 > libraries: > > boost-cropped givaro libm4ri mpir ratpoints > cliquer gsl libpng ntl > eclib iml linbox pari singular > ecm lcalc mpfi polybori symmetrica > flint libfplll mpfr pynac zn_poly > > This function in setup.py is a Python function, and it can do > *anything* it wants. distutils/setuptools/etc. are irrelevant!! In ... > --- > > Anyway, since this thread sort of ended with some major misconceptions > that the setuptools weirdness was a serious issue, I wanted to correct > this misconception. Yes, there were some misconceptions. My concern is that maintaining the ability to build all these 24 C/C++ libraries currently is quite some effort: on quite a variety of systems, with several toolchains, with $SAGE_ROOT as prefix, and all the necessary patches here and there ... And new OS versions do come out (Fedora 14, OS X 10.7, OpenSuse 12.0, ...), as well as new versions of one of these 24 libraries themselves every now and then. Thanks for posting the "setup.py" script, it clearly illuminates that we were talking at cross-purposes (I hope I looked up that wording correctly :-)). I think it has become clear now, that without some help of tooling (scripts, or whatever), the situation will only get worse (Leif already pointed out issues). Why not "outsource" this business, including the tooling itself? Then the "build_package()" function in your "setup.py" would essentially just be a one-liner: emerge name (using ebuilds and portage from Gentoo) or maybe, say pacman name (using pkgbuilds and pacman from Arch Linux), and e.g. inter- dependencies would be resolved automatically. Using a such a full- grown tooling for "just" 24 packages (i.e C/C++ libraries) might seem overkill at first sight. But I don't think so, to the contrary, the overhead (portage e.g. is less than a megabyte compressed) is pretty low, and to me, seems well worth it. (And I hope we can agree that setuptools/distutils is not suited for this.) But that should deserve its own thread, so I stop here. > > Another point I think is interesting is that the Sage library itself > seriously depends on the above 24 C/C++ libraries, which have little > or nothing to do with Python2 versus Python3, plus a very small number > of Python libraries: numpy, matplotlib, networkx. Sage uses scipy, > cvxopt, etc., a tiny, tiny bit, but nothing serious. Even matplotlib > is *only* used to draw pictures. Thus if we wanted a Python3 version > of the Sage library itself, if we had a library like I describe above, > this would only require a Python3 version of numpy and networkx, plus > the work of porting the Sage library itself. This doesn't sound so > far off, since there already is a Python3 version of numpy. > I'm all for slicing up the current rather monolithic Sage distribution into smaller, more manageable parts. Having an independent "Sage core", the transition to Python 3 will certainly be less painful (the question is not whether, but only when we'll do this). It is even thinkable to have some parts using Python 3 (the Sage core, say) and some parts still using Python 2 (SageNB? or which parts of the Sage distribution were relying on that old version of Twisted?) at one and the same time, even in officially shipped distributions ... Cheers, Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 1 Nov., 17:16, William Stein wrote: > On Mon, Nov 1, 2010 at 7:34 AM, leif wrote: > > On 1 Nov., 07:51, William Stein wrote: > >> This post is about: > > >> (1) Concern about distutils/setuptools/etc., is misplaced. > >> (2) Python3 and librarifying Sage. > > >> First, all this discussion about distutils/setuptools/david > >> cournapeau, etc., is actually mostly IRRELEVANT to making the core > >> Sage library into a standalone library. > > > That's an argument (honestly). Distutils/Setuptools is broken (or > > weird), so we don't [have to] use it. > > Not very "pythonic" though, I guess... ;-) (but nobody should be > > forced to drive with broken wheels) > > I don't quite agree with this interpretation. Even if > setuptools/distutils were "perfect" they would not be the right tool > for building those 24 libraries. The right tool is a Python script > that calls the native build system on each of those 24 libraries > (e.g., which is autoconf, perl, etc.). Well, if they /were/ "perfect", one could perhaps use them for such as well. (Developers, either upstream or we, could also ship a setup.py or similar as an alternative to "configure/make/make install" or spkg- install. I first thought you also aimed at the latter.) My point was it previously wasn't clear to me that you intended to keep / use the spkg-install scripts of the spkgs. > > I'd prefer having plain text files rather than a pickled build_db. > > Thank you for looking at the code. > > It is just a pickled pure python dictionary, which is flexible and > easy to work with. Easy to work with from Python. I like having human-readable (and writable) configuration etc. files in a format that's pretty generic to be managed / processed by shell and other scripts / programs as well. But that's UNIX philosophy. (And I didn't mean XML.) > > Adding (formal) dependency specifications to the spkgs (a file, say, > > spkg-deps in each spkg) would be a step forward, too, such that we can > > also *generate* the "real Makefile" spkg/standard/deps in the > > traditional build process. > > There's no deps file in what I'm doing. I guess /not yet/, since I think it is a prototype. > Also, among these 24 > libraries the dependencies are nearly trivial. Basically, many depend > on MPIR, some on NTL, and beyond that there is nothing (?). :-) No reason to hard-code them, especially if we could use such deps elsewhere, too, even for documentation. And as always, they are subject to change. Having them attached to the spkg files rather than in a manually maintained, separate "global" file (currently not even under revision control, cf. #9433) is less error-prone and aids modularity. > > (In addition, a lot of what's currently performed in every spkg- > > install could be factored out.) > > True, but orthogonal. Yes, but (IMHO closely) related, to "both ways" of installing Sage. -Leif -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 11/1/10 1:33 PM, William Stein wrote: My plan for migrating the Sage notebook to not use twisted anymore is to switch to Flask (http://flask.pocoo.org/). Flask is a small "microframework", but it only works with Python 2.x, and they have no plans at present to support Python 3.x. Evidently, they believe that there is no good mod_wsgi support in Python 3.x yet. Citation: http://flask.pocoo.org/docs/installation/ "At the time of writing, the WSGI specification is not yet finalized for Python 3, so Flask cannot support the 3.x series of Python." So it sounds like it's just a matter of time... Jason -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: All of sage as a python library
On 11/1/10 8:09 PM, Jason Grout wrote: On 11/1/10 1:33 PM, William Stein wrote: My plan for migrating the Sage notebook to not use twisted anymore is to switch to Flask (http://flask.pocoo.org/). Flask is a small "microframework", but it only works with Python 2.x, and they have no plans at present to support Python 3.x. Evidently, they believe that there is no good mod_wsgi support in Python 3.x yet. Citation: http://flask.pocoo.org/docs/installation/ "At the time of writing, the WSGI specification is not yet finalized for Python 3, so Flask cannot support the 3.x series of Python." So it sounds like it's just a matter of time... And an even better citation: http://flask.pocoo.org/docs/foreword/#the-status-of-python-3 Jason -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: All of sage as a python library
On Wed, Oct 27, 2010 at 3:48 AM, Volker Braun wrote: > Just to clarify, are we talking about different namespaces > > from sagecore.rings import Integers > from sagemain.modular.all import euler_phi > from sagecombinat.combinat import choose_nk > > This seems a bit unwieldy. I'm not talking about that... yet. But even if I were, for an end sage user, there's nothing stopping us from importing everything into sage exactly as now. People seem to have a big concern that sage-as-it-is-now would cease to exist were we to do what I'm describing. This is a misplaced fear. In fact, it would just change how Sage itself is put together, and make the work that people put into Sage available to a vastly wider community of users. -- William > On the other hand, if Sage pulls everything > into sage.* then how do I know which library I'm using. On a related > note, can the sagecore documentation link to the main sage library > documentation? > > Volker > > > On Oct 27, 2:44 am, William Stein wrote: >> Hi, >> >> When I started Sage I viewed it as a distribution of a bunch of math >> software, and Python as just the interpreter language I happen to use >> at the time. I didn't even know if using Python as the language would >> last. However, it's also possible to think of Sage as a Python >> library. >> >> Anyway, it has occurred to me (a few times, and again recently) that >> it would be possible to make much of the Sage distribution, without >> Python of course, into a Python library. What I mean is the >> following. You would have a big Python library called "sagemath", >> say, and inside of that would be a huge HG repository. In that >> repository, one would check in the source code for many of the >> standard Sage spkg's... e.g., GAP, Pari, etc. When you type >> >> python setup.py install >> >> then GAP, Pari, etc., would all get built, controlled by some Python >> scripts, then installed as package_data in the sagemath directory of >> /site-packages/. >> >> From a technical perspective, I don't see any reason why this couldn't >> be made to work. HG can handle this much data, and "python setup.py >> install" can do anything. It does lead to a very different way of >> looking at Sage though, and it could help untangle things in >> interesting ways. >> >> (1) Have a Python library called "sagecore", which is just the most >> important standard spkg's (e.g., Singular, PARI, etc.), perhaps >> eventually built *only* as shared object libraries (no standalone >> interpreters). >> >> (2) Have a Python library which is the current Sage library (we >> already have this), and which can be installed assuming sagecore is >> installed. >> >> (3) Have other Python libraries (like >> psage:http://code.google.com/p/purplesage/source/browse/), which depend on >> (2). Maybe a lot of the "sage-combinat" code could also be moved to >> such a library, so they can escape the "combinat patch queue" madness. >> Maybe many other research groups in algebraic topology, differential >> geometry, special functions, etc., will start developing such >> libraries... on their own, and share them with the community (but >> without having to deal directly with the sage project until they want >> to). >> >> To emphasize (3), when people want to write a lot of mathematics code >> in some area, e.g., differential geometry, they would just make a new >> library that depends on Sage (the library in (2)). We do the work >> needed to make it easy for people to write code outside of the Sage >> library, which depends on Sage. Especially writing Cython code like >> this can be difficult and confusing, and we don't explain it all in >> any Sage documentation. It actually took me quite a while to figure >> out how to do it today (with psage). >> >> The core Sage library (2) above would continue to have a higher and >> higher level of code review, tough referee process etc. However, the >> development models for (3) would be up to the authors of those >> libraries. >> >> The above is already how the ecosystem with Python >> (http://pypi.python.org/pypi), Perl (http://www.cpan.org/), R, etc., >> work. Fortunately, Python has reasonably good support already for >> this. >> >> I think without a shift in this direction, Sage is going to be very >> frustrating for people writing research oriented code. >> >> Fortunately, it's possible to do everything I'm describing above >> without disturbing the mainline Sage project itself, at least for >> now. >> >> -- William >> >> -- >> William Stein >> Professor of Mathematics >> University of Washingtonhttp://wstein.org > > -- > To post to this group, send an email to sage-devel@googlegroups.com > To unsubscribe from this group, send an email to > sage-devel+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- William Stein Professor of Mathematics University of Washington
Re: [sage-devel] Re: All of sage as a python library
On Tue, Oct 26, 2010 at 8:54 PM, Dima Pasechnik wrote: > Being able to get Sage as a part of PyPI would be great! > Taking into account how many of Sage spkgs are there, e.g. cython, > scipy, networkx, cvxopt, > this looks like the right way of factoring > out components that are just packaged into Sage. I actually started doing something, which is at http://sage.math.washington.edu/home/wstein/sagecore/ All I did was make a list of all the C/C++ libraries that come with Sage and which the core Sage library depends on due to Cython code that uses these libraries. These are also libraries that aren't "bog standard". Here is the list: cliquer eclib ecm flint givaro gsl iml lcalc libfplll libm4ri linbox mpfi mpfr mpir ntl pari polybori pynac ratpoints singular symmetrica zn_poly For each in the list, I copied (via a script I wrote), the spkg over to a directory "sagecore/packages", extracted the package, got rid of the explicit version number from it (it's in the SPKG.txt file anyways), got rid of all the .hg repos in there, and the result is at http://sage.math.washington.edu/home/wstein/sagecore/sagecore.tar.bz2 You can browse (for a limited time) the hg repo at http://sage.math.washington.edu:8010/file/d569a5041000/packages The next step is to write a script that goes through and builds all the above packages into a directory. Then make it so that gets installed into a standard Python package data directory, and then build the Sage library on top of this. William > At the moment just keeping apace with the latter > components is taking a large chunk of developers time. This is very, very true, and your point about cvxopt later on nicely emphasizes this. > > To begin with, I imagine one can look into ways PyPI manages > dependencies, etc. > I understand there is a mechanism that allows for pinning of > particular versions, etc... Yep, and also easily packaging up certain versions of all dependencies with a package (sagenb uses this, actually). PyPI rocks. William > > > > On Oct 27, 9:44 am, William Stein wrote: >> Hi, >> >> When I started Sage I viewed it as a distribution of a bunch of math >> software, and Python as just the interpreter language I happen to use >> at the time. I didn't even know if using Python as the language would >> last. However, it's also possible to think of Sage as a Python >> library. >> >> Anyway, it has occurred to me (a few times, and again recently) that >> it would be possible to make much of the Sage distribution, without >> Python of course, into a Python library. What I mean is the >> following. You would have a big Python library called "sagemath", >> say, and inside of that would be a huge HG repository. In that >> repository, one would check in the source code for many of the >> standard Sage spkg's... e.g., GAP, Pari, etc. When you type >> >> python setup.py install >> >> then GAP, Pari, etc., would all get built, controlled by some Python >> scripts, then installed as package_data in the sagemath directory of >> /site-packages/. >> >> From a technical perspective, I don't see any reason why this couldn't >> be made to work. HG can handle this much data, and "python setup.py >> install" can do anything. It does lead to a very different way of >> looking at Sage though, and it could help untangle things in >> interesting ways. >> >> (1) Have a Python library called "sagecore", which is just the most >> important standard spkg's (e.g., Singular, PARI, etc.), perhaps >> eventually built *only* as shared object libraries (no standalone >> interpreters). >> >> (2) Have a Python library which is the current Sage library (we >> already have this), and which can be installed assuming sagecore is >> installed. >> >> (3) Have other Python libraries (like >> psage:http://code.google.com/p/purplesage/source/browse/), which depend on >> (2). Maybe a lot of the "sage-combinat" code could also be moved to >> such a library, so they can escape the "combinat patch queue" madness. >> Maybe many other research groups in algebraic topology, differential >> geometry, special functions, etc., will start developing such >> libraries... on their own, and share them with the community (but >> without having to deal directly with the sage project until they want >> to). >> >> To emphasize (3), when people want to write a lot of mathematics code >> in some area, e.g., differential geometry, they would just make a new >> library that depends on Sage (the library in (2)). We do the work >> needed to make it easy for people to write code outside of the Sage >> library, which depends on Sage. Especially writing Cython code like >> this can be difficult and confusing, and we don't explain it all in >> any Sage documentation. It actually took me quite a while to figure >> out how to do it today (with psage). >> >> The core Sage library (2) above would continue to have a higher and >> higher level of code review, tough referee pro
Re: [sage-devel] Re: All of sage as a python library
On Thu, Oct 28, 2010 at 11:02 PM, Dima Pasechnik wrote: > > > On Oct 28, 4:23 am, "Georg S. Weber" > wrote: >> > (1) Have a Python library called "sagecore", which is just the most >> > important standard spkg's (e.g., Singular, PARI, etc.), perhaps >> > eventually built *only* as shared object libraries (no standalone > [...] >> >> I just can't believe David Cournapeau would go this way, if "python >> setup.py install" could do "anything", or if Python resp. its ecosysem >> (PyPI, ...) already had reasonably good support for the needs of the >> NumPy project. >> > > the point that David Cournapeau makes is about distutils/setuputils > getting > too messy, too procedural, as opposed to declarative. His point about > the need > of a better packaging system is not about the lack of power in the > present system, > it's about lack of structure and abundance of general mess... This post is about: (1) Concern about distutils/setuptools/etc., is misplaced. (2) Python3 and librarifying Sage. First, all this discussion about distutils/setuptools/david cournapeau, etc., is actually mostly IRRELEVANT to making the core Sage library into a standalone library. The way it would work is this: 1. You type "python setup.py develop" (or possibly "python setup.py install"). 2. A function in setup.py builds all the non-standard C/C++ libraries that the core Sage library depends on, which is the following 24 libraries: boost-cropped givaro libm4ri mpirratpoints cliquer gsl libpng ntl eclib iml linbox parisingular ecm lcalc mpfipolyborisymmetrica flint libfplllmpfrpynac zn_poly This function in setup.py is a Python function, and it can do *anything* it wants. distutils/setuptools/etc. are irrelevant!!In fact, this can just be a very simple version of the current Sage build system, and we can just include the 24 Sage packages corresponding the above-listed 24 libraries basically as is. Just for fun, I tried this and wrote a sample setup.py sort of illustrating what I mean (and ran it, and it works, but you can't, since of course it needs the source files. I'll post more later.). When I did this, by the way, and deleted the .a files, leaving just the shared libraries, it only took about 25MB compressed -- pretty interesting. 3. After the C/C++ libraries have all been built, then the regular Sage library gets built, using some slight variation of the current build scripts. --- Anyway, since this thread sort of ended with some major misconceptions that the setuptools weirdness was a serious issue, I wanted to correct this misconception. Another point I think is interesting is that the Sage library itself seriously depends on the above 24 C/C++ libraries, which have little or nothing to do with Python2 versus Python3, plus a very small number of Python libraries: numpy, matplotlib, networkx. Sage uses scipy, cvxopt, etc., a tiny, tiny bit, but nothing serious. Even matplotlib is *only* used to draw pictures. Thus if we wanted a Python3 version of the Sage library itself, if we had a library like I describe above, this would only require a Python3 version of numpy and networkx, plus the work of porting the Sage library itself. This doesn't sound so far off, since there already is a Python3 version of numpy. -- William -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org setup.py Description: Binary data
Re: [sage-devel] Re: All of sage as a python library
On Mon, Nov 1, 2010 at 7:34 AM, leif wrote: > On 1 Nov., 07:51, William Stein wrote: >> This post is about: >> >> (1) Concern about distutils/setuptools/etc., is misplaced. >> (2) Python3 and librarifying Sage. >> >> First, all this discussion about distutils/setuptools/david >> cournapeau, etc., is actually mostly IRRELEVANT to making the core >> Sage library into a standalone library. > > That's an argument (honestly). Distutils/Setuptools is broken (or > weird), so we don't [have to] use it. > Not very "pythonic" though, I guess... ;-) (but nobody should be > forced to drive with broken wheels) I don't quite agree with this interpretation. Even if setuptools/distutils were "perfect" they would not be the right tool for building those 24 libraries. The right tool is a Python script that calls the native build system on each of those 24 libraries (e.g., which is autoconf, perl, etc.). > > >> [...] >> 2. A function in setup.py builds all the non-standard C/C++ libraries >> that the core Sage library depends on, which is the following 24 >> libraries: >> >> boost-cropped givaro libm4ri mpir ratpoints >> cliquer gsl libpng ntl >> eclib iml linbox pari singular >> ecm lcalc mpfi polybori symmetrica >> flint libfplll mpfr pynac zn_poly >> [...] > > I'd prefer having plain text files rather than a pickled build_db. Thank you for looking at the code. It is just a pickled pure python dictionary, which is flexible and easy to work with. > Adding (formal) dependency specifications to the spkgs (a file, say, > spkg-deps in each spkg) would be a step forward, too, such that we can > also *generate* the "real Makefile" spkg/standard/deps in the > traditional build process. There's no deps file in what I'm doing. Also, among these 24 libraries the dependencies are nearly trivial. Basically, many depend on MPIR, some on NTL, and beyond that there is nothing (?). > (In addition, a lot of what's currently performed in every spkg- > install could be factored out.) True, but orthogonal. > > > -Leif > >> [...] Thus if we wanted a Python3 version >> of the Sage library itself, if we had a library like I describe above, >> this would only require a Python3 version of numpy and networkx, plus >> the work of porting the Sage library itself. This doesn't sound so >> far off, since there already is a Python3 version of numpy. > > -- > To post to this group, send an email to sage-devel@googlegroups.com > To unsubscribe from this group, send an email to > sage-devel+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- William Stein Professor of Mathematics University of Washington http://wstein.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: All of sage as a python library
On Mon, Nov 1, 2010 at 9:48 AM, Georg S. Weber wrote: > >> >> This post is about: >> >> (1) Concern about distutils/setuptools/etc., is misplaced. >> (2) Python3 and librarifying Sage. >> >> First, all this discussion about distutils/setuptools/david >> cournapeau, etc., is actually mostly IRRELEVANT to making the core >> Sage library into a standalone library. The way it would work is >> this: >> >> 1. You type "python setup.py develop" (or possibly "python setup.py >> install"). >> >> 2. A function in setup.py builds all the non-standard C/C++ libraries >> that the core Sage library depends on, which is the following 24 >> libraries: >> >> boost-cropped givaro libm4ri mpir ratpoints >> cliquer gsl libpng ntl >> eclib iml linbox pari singular >> ecm lcalc mpfi polybori symmetrica >> flint libfplll mpfr pynac zn_poly >> >> This function in setup.py is a Python function, and it can do >> *anything* it wants. distutils/setuptools/etc. are irrelevant!! In > > ... > >> --- >> >> Anyway, since this thread sort of ended with some major misconceptions >> that the setuptools weirdness was a serious issue, I wanted to correct >> this misconception. > > Yes, > > there were some misconceptions. My concern is that maintaining the > ability to build all these 24 C/C++ libraries currently is quite some > effort: on quite a variety of systems, with several toolchains, with > $SAGE_ROOT as prefix, and all the necessary patches here and there ... > > And new OS versions do come out (Fedora 14, OS X 10.7, OpenSuse > 12.0, ...), as well as new versions of one of these 24 libraries > themselves every now and then. > > Thanks for posting the "setup.py" script, it clearly illuminates that > we were talking at cross-purposes (I hope I looked up that wording > correctly :-)). I think it has become clear now, that without some > help of tooling (scripts, or whatever), the situation will only get > worse (Leif already pointed out issues). Why not "outsource" this > business, including the tooling itself? Then the "build_package()" > function in your "setup.py" would essentially just be a one-liner: > > emerge name > > (using ebuilds and portage from Gentoo) or maybe, say > > pacman name > > (using pkgbuilds and pacman from Arch Linux), and e.g. inter- > dependencies would be resolved automatically. Using a such a full- > grown tooling for "just" 24 packages (i.e C/C++ libraries) might seem > overkill at first sight. But I don't think so, to the contrary, the > overhead (portage e.g. is less than a megabyte compressed) is pretty > low, and to me, seems well worth it. (And I hope we can agree that > setuptools/distutils is not suited for this.) > > But that should deserve its own thread, so I stop here. I again think that your above remarks are totally completely orthogonal to what I'm proposing. It may be that maintaining those 24 packages is a lot of work, but that's work that is not relevant to making sage into a standalone library. It's work that is *already* been done (and being done) for the standalone version of Sage. If (and only if) Sage were to switch to a different build system, then the standalone sage library could just adapt to that. >> Another point I think is interesting is that the Sage library itself >> seriously depends on the above 24 C/C++ libraries, which have little >> or nothing to do with Python2 versus Python3, plus a very small number >> of Python libraries: numpy, matplotlib, networkx. Sage uses scipy, >> cvxopt, etc., a tiny, tiny bit, but nothing serious. Even matplotlib >> is *only* used to draw pictures. Thus if we wanted a Python3 version >> of the Sage library itself, if we had a library like I describe above, >> this would only require a Python3 version of numpy and networkx, plus >> the work of porting the Sage library itself. This doesn't sound so >> far off, since there already is a Python3 version of numpy. >> > > I'm all for slicing up the current rather monolithic Sage distribution > into smaller, more manageable parts. Having an independent "Sage > core", the transition to Python 3 will certainly be less painful (the > question is not whether, but only when we'll do this). It is even > thinkable to have some parts using Python 3 (the Sage core, say) and > some parts still using Python 2 (SageNB? or which parts of the Sage > distribution were relying on that old version of Twisted?) at one and > the same time, even in officially shipped distributions ... I really, really hope that having to ship both never happens, but yes, it is technically possible. My plan for migrating the Sage notebook to not use twisted anymore is to switch to Flask (http://flask.pocoo.org/). Flask is a small "microframework", but it only works with Python 2.x, and they have no plans at present to support
