Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 9:32 PM, William Stein wrote: > On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer > wrote: >> On 2016-04-14 20:35, William Stein wrote: >>> >>> However I'm only now starting to complain loudly and >>> repeatedly just because I'm seeing such a huge wasted opportunity. >> >> >> Instead of complaining, why don't you put together a more concrete and >> technical proposal of what you actually want? Because this thread is getting >> really long with a low fact/opinion ratio. Surely by far not enough facts to >> convince me. >> >> I also see a lot of discussion in this thread about general issues with Sage >> and Sage development which are unrelated to modularisation. So, when you do >> write that proposal, be sure to stick to problems which are really caused by >> the monolithic nature of Sage. > > Just to be clear: I personally won't be writing any such thing anytime > in the near future. > > I guess my conclusion for now from this thread is not that I'm trying > to convince you of anything. However, as a said before, my > recommendation to anybody considering writing code for Sage is, to > quote Francesco Biscani, to "move out of the reality distortion field > and apply normal software development practices." Here's a concrete > example of this happening: > >https://github.com/haochenuw/sage_modabvar I considered doing something like this myself once, but it seemed nearly impossible. Instead I spent quite some time developing my own symbolic manipulation and linear algebra code for a teaching/exploration platform for quantum information and cryptography that I worked on. Like most good software it did leverage a lot of work from existing libraries such as Numpy and SciPy, and also took a lot of advantage of the IPython notebook for the latex and graphics support. I looked into building it on top of sage but it seemed untenable. Installing sage was not easy for me, a professional, and I wasn't about to impose it on anyone else who wanted to try out my software. :( Good to see that at least some people are trying to do this. If you can manage to use Sage as a library all power to you--literally--there's a lot of power in there. But I wasn't about to try to put my stuff directly into Sage, and even if that did seem viable I'm not so sure that would have been appropriate to, since its goal was somewhat orthogonal--a big part of it for example was a DSL for quantum circuits. Wouldn't have made sense as "core" functionality in a mathematics library. Or so I thought. Though seeing some of the other stuff in there (which is not *bad* or anything) maybe I'm wrong. Though as it stands it seems like it was a good decision not to integrate with sage and become trapped in its reality distortion field that interacts poorly with the rest of the scientific Python community. [In fairness I've since moved on to other things and let that project languish so it's hard to say whether or not its success would have been effected one way or the other.] Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thursday, April 14, 2016 at 10:50:32 PM UTC+1, Chris Swierczewski wrote: > > (Disclaimer: I haven't yet read this entire thread when writing my > response.) > > Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I >> know. I recently ran into him and he told me that he had spent years >> writing this package as a standard Python package depending on sympy >> (mainly) and not Sage. However, it was getting quite frustrating to >> finish abelfunctions, due to him needing things that are in Sage (I >> think things like interval arithmetic?). He didn't want to develop >> his code in a way that depended on Sage, because he wanted it to be >> more widely available, and he clearly didn't see the value in having >> the code packaged as part of Sage for only a few small things.This >> is a great example of how our centralized monolithic approach is >> overall not the best for the community at large. >> >> I've cc'd Chris in case he wants to clarify this, since I might not >> understand. >> > > I remember discussions of this "modularized Sage" proposal way back when I > was an undergraduate so I don't think it's anything new. In fact, hearing > William talk about these issues back then is, in part, what prompted me to > design my Sage package the way I did. > > That being said, I can only really comment on my own experience developing > Abelfunctions (http://github.com/abelfunctions/ableufncitons). As William > mentioned, I started writing Abelfunctions such that only depended on very > common core Python packages. Anaconda is very popular in the applied > mathematics community and many of the key scientific computing packages I > needed (e.g. Numpy, Scipy, networkx) are included in Anaconda as well as in > Sage. This design seemed like a good idea at the time since my code could > reach as many people as possible. > > However, I slowly realized that Sympy's limitations in *efficiently* > performing exact arithmetic with polynomial roots would prevent me from > completing my software. After attempting to basically write my own version > of QQbar in Sympy I made the very difficult decision, after four+ years of > my Ph.D. writing Sympy workarounds, to completely rewrite my code "using > Sage". I'm glad I did even though it added time to my Ph.D. --- a very > small amount of time compared to the amount wasted trying to get Sympy to > do what I wanted. So I am definitely glad that something like Sage is > around to provide those open source tools I needed. > > I didn't pursue writing Abelfunctions into Sage or even making > Abelfunctions an "optional" or "experimental" listed spkg for several > reasons: > > 1. *It is still in very active development* --- My quick (read: several > month) total rewrite of Abelfunctions to use Sage reintroduced bugs that I > squashed in the Python version. This is somewhat evidenced in the GitHub > issues page. Because I'm constantly fixing and adding things as I finish my > thesis it seems bothersome to have to push a new spkg after every commit. > (Sure, I can write a script to try and do that.) > > 2. *It is a niche product* --- The mathematical communities I work with > know about Abelfunctions and they have spread the word of its existence to > those who might be interested. That's great. But I never assumed that it > should be a standard part of Sage. Sage already takes too much time to > compile so why bloat it with something that <1% of Sage users would ever > touch? (Though, my implementation of Puiseux series in Abelfunctions is now > making it's way into the Sage codebase.) > > 3. *GitHub is my resume* --- Headhunters have seen that I write code. I > doubt they scan trac.sagemath as much as they do GitHub. Being able to put > my code up on GitHub has made it clearer to others and to myself what my > contribution to the math community has been. Not that I'm against sharing > in an open-source project but to have Abelfunctions Trac tickets buried in > the mound of other Sage issues didn't seem like a clear way to get people > to work on this small project at the time. > Sage is on github: https://github.com/sagemath/sage/graphs/contributors lists contributors to the master (not very accurately though - but perhaps it can, and should, be fixed) > > Overall, developing Abelfunctions as an "external" Sage package has been > pretty effective. I have some minor complaints, including the fact that the > standard "$ sage setup.py install" doesn't seem to work on all platforms > and trying to get TravisCI to download and install Sage before being able > to test my code is a bit of a pain. If people who want to use it are having > trouble finding it then it's my fault, not Sage's, since I suck at > advertising myself. (Writing this post about my own code was difficult > enough.) I think if enough people write packages this way we will see what > works and what doesn't thus putting the whole "let's
Re: [sage-devel] Re: how we develop sage
On 2016-04-14 21:32, William Stein wrote: So I wonder if you think that, say, elliptic curves should be a separate package too? I'm not lowering the fact/opinion ratio further... I would at least help to understand what exactly it is that you are proposing. You keep using the words "normal software development practices" but that's so vague it's not helping. And I cannot read your mind... -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> > > Chris started with something here: > > https://gist.github.com/cswiercz/c632d920565a2da519b73bd2b79d7920 > > Honestly, this could be of great use also for something intended 'immediately' for the Sage library too, very nicely put together thus far. > Rather than a section of the sage reference manual, I would prefer a link > (in the sage reference manual, etc.) to a wiki page on github, e.g., there > is > a Sage wiki on github here, which we could use: > > https://github.com/sagemath/sage/wiki > > > This seems like a pretty good idea. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 6:38 PM, Kwankyu Lee wrote: > > > On Friday, April 15, 2016 at 6:54:44 AM UTC+9, kcrisman wrote: >> >> >>> > These packages are nearly impossible to found from the sagemath >>> > website! >>> >>> Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I >>> know. I recently ran into him and he told me that he had spent years >> >> >> This is a good point; we could use more infrastructure for supporting the >> advertising of packages of this kind. > > > And it would be nice to include some guidelines for development of these > external Sage packages in the Sage developer manual. The topics would > include > > - Tips for better integration with Sage core > - Standard (or recommended) way to install the package into Sage (before > "make" and after "make") > - Proper versioning > - How to be advertised in the SageMath website Chris started with something here: https://gist.github.com/cswiercz/c632d920565a2da519b73bd2b79d7920 Rather than a section of the sage reference manual, I would prefer a link (in the sage reference manual, etc.) to a wiki page on github, e.g., there is a Sage wiki on github here, which we could use: https://github.com/sagemath/sage/wiki -- William -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 15, 2016 at 8:46:44 AM UTC+9, Chris Swierczewski wrote: > > I felt like writing this up: > > https://gist.github.com/cswiercz/c632d920565a2da519b73bd2b79d7920 > > Please suggest improvements and corrections. > Sorry. I didn't see your post before I wrote mine :-) That is a nice start. I hope that it would grow and eventually make its way into Sage developer manual. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Friday, April 15, 2016 at 6:54:44 AM UTC+9, kcrisman wrote: > > > > These packages are nearly impossible to found from the sagemath website! >> >> Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I >> know. I recently ran into him and he told me that he had spent years >> > > This is a good point; we could use more infrastructure for supporting the > advertising of packages of this kind. > And it would be nice to include some guidelines for development of these external Sage packages in the Sage developer manual. The topics would include - Tips for better integration with Sage core - Standard (or recommended) way to install the package into Sage (before "make" and after "make") - Proper versioning - How to be advertised in the SageMath website etc. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
I felt like writing this up: https://gist.github.com/cswiercz/c632d920565a2da519b73bd2b79d7920 Please suggest improvements and corrections. -- Chris -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Thursday, April 14, 2016 at 8:35:29 PM UTC+2, William wrote: > > apply normal software development practices!! You mean like in the Linux kernel? A project which, by the way, is a) extremely anal about having a stable API (=posix) and b) very clear about what is not going to be stable. Or do you mean split up things but with no specified API between modules, no tests on said non-existing API, and no independent testsuites whatsoever? And flush documentation down the toilet while we are at it so it builds faster? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> > These packages are nearly impossible to found from the sagemath website! > > Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I > know. I recently ran into him and he told me that he had spent years > This is a good point; we could use more infrastructure for supporting the advertising of packages of this kind. > writing this package as a standard Python package depending on sympy > (mainly) and not Sage. However, it was getting quite frustrating to > finish abelfunctions, due to him needing things that are in Sage (I > think things like interval arithmetic?). He didn't want to develop > his code in a way that depended on Sage, because he wanted it to be > more widely available, and he clearly didn't see the value in having > the code packaged as part of Sage for only a few small things.This > is a great example of how our centralized monolithic approach is > overall not the best for the community at large. > > I don't know whether this relevant, but it reminds me of people who write code depending on Mathematica (for which there are many such packages, not least of which knot stuff)? Obviously it won't end up in Mma proper, but people will depend upon Mma. And such packages definitely seem to get a heavy amount of use. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
(Disclaimer: I haven't yet read this entire thread when writing my response.) Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I > know. I recently ran into him and he told me that he had spent years > writing this package as a standard Python package depending on sympy > (mainly) and not Sage. However, it was getting quite frustrating to > finish abelfunctions, due to him needing things that are in Sage (I > think things like interval arithmetic?). He didn't want to develop > his code in a way that depended on Sage, because he wanted it to be > more widely available, and he clearly didn't see the value in having > the code packaged as part of Sage for only a few small things.This > is a great example of how our centralized monolithic approach is > overall not the best for the community at large. > > I've cc'd Chris in case he wants to clarify this, since I might not > understand. > I remember discussions of this "modularized Sage" proposal way back when I was an undergraduate so I don't think it's anything new. In fact, hearing William talk about these issues back then is, in part, what prompted me to design my Sage package the way I did. That being said, I can only really comment on my own experience developing Abelfunctions (http://github.com/abelfunctions/ableufncitons). As William mentioned, I started writing Abelfunctions such that only depended on very common core Python packages. Anaconda is very popular in the applied mathematics community and many of the key scientific computing packages I needed (e.g. Numpy, Scipy, networkx) are included in Anaconda as well as in Sage. This design seemed like a good idea at the time since my code could reach as many people as possible. However, I slowly realized that Sympy's limitations in *efficiently* performing exact arithmetic with polynomial roots would prevent me from completing my software. After attempting to basically write my own version of QQbar in Sympy I made the very difficult decision, after four+ years of my Ph.D. writing Sympy workarounds, to completely rewrite my code "using Sage". I'm glad I did even though it added time to my Ph.D. --- a very small amount of time compared to the amount wasted trying to get Sympy to do what I wanted. So I am definitely glad that something like Sage is around to provide those open source tools I needed. I didn't pursue writing Abelfunctions into Sage or even making Abelfunctions an "optional" or "experimental" listed spkg for several reasons: 1. *It is still in very active development* --- My quick (read: several month) total rewrite of Abelfunctions to use Sage reintroduced bugs that I squashed in the Python version. This is somewhat evidenced in the GitHub issues page. Because I'm constantly fixing and adding things as I finish my thesis it seems bothersome to have to push a new spkg after every commit. (Sure, I can write a script to try and do that.) 2. *It is a niche product* --- The mathematical communities I work with know about Abelfunctions and they have spread the word of its existence to those who might be interested. That's great. But I never assumed that it should be a standard part of Sage. Sage already takes too much time to compile so why bloat it with something that <1% of Sage users would ever touch? (Though, my implementation of Puiseux series in Abelfunctions is now making it's way into the Sage codebase.) 3. *GitHub is my resume* --- Headhunters have seen that I write code. I doubt they scan trac.sagemath as much as they do GitHub. Being able to put my code up on GitHub has made it clearer to others and to myself what my contribution to the math community has been. Not that I'm against sharing in an open-source project but to have Abelfunctions Trac tickets buried in the mound of other Sage issues didn't seem like a clear way to get people to work on this small project at the time. Overall, developing Abelfunctions as an "external" Sage package has been pretty effective. I have some minor complaints, including the fact that the standard "$ sage setup.py install" doesn't seem to work on all platforms and trying to get TravisCI to download and install Sage before being able to test my code is a bit of a pain. If people who want to use it are having trouble finding it then it's my fault, not Sage's, since I suck at advertising myself. (Writing this post about my own code was difficult enough.) I think if enough people write packages this way we will see what works and what doesn't thus putting the whole "let's refactor Sage to make it modular" into a more concrete setting. -- Chris -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://grou
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 12:41 PM, Vincent Delecroix <20100.delecr...@gmail.com> wrote: > On 14/04/16 16:32, William Stein wrote: >> >> On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer >> wrote: >>> >>> On 2016-04-14 20:35, William Stein wrote: However I'm only now starting to complain loudly and repeatedly just because I'm seeing such a huge wasted opportunity. >>> >>> >>> >>> Instead of complaining, why don't you put together a more concrete and >>> technical proposal of what you actually want? Because this thread is >>> getting >>> really long with a low fact/opinion ratio. Surely by far not enough facts >>> to >>> convince me. >>> >>> I also see a lot of discussion in this thread about general issues with >>> Sage >>> and Sage development which are unrelated to modularisation. So, when you >>> do >>> write that proposal, be sure to stick to problems which are really caused >>> by >>> the monolithic nature of Sage. >> >> >> Just to be clear: I personally won't be writing any such thing anytime >> in the near future. >> >> I guess my conclusion for now from this thread is not that I'm trying >> to convince you of anything. However, as a said before, my >> recommendation to anybody considering writing code for Sage is, to >> quote Francesco Biscani, to "move out of the reality distortion field >> and apply normal software development practices." Here's a concrete >> example of this happening: >> >> https://github.com/haochenuw/sage_modabvar > > > Or > > https://github.com/abelfunctions/abelfunctions > https://github.com/coulbois/sage-train-track > > These packages are nearly impossible to found from the sagemath website! Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I know. I recently ran into him and he told me that he had spent years writing this package as a standard Python package depending on sympy (mainly) and not Sage. However, it was getting quite frustrating to finish abelfunctions, due to him needing things that are in Sage (I think things like interval arithmetic?). He didn't want to develop his code in a way that depended on Sage, because he wanted it to be more widely available, and he clearly didn't see the value in having the code packaged as part of Sage for only a few small things.This is a great example of how our centralized monolithic approach is overall not the best for the community at large. I've cc'd Chris in case he wants to clarify this, since I might not understand. William > > Vincent > > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 14/04/16 16:32, William Stein wrote: On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer wrote: On 2016-04-14 20:35, William Stein wrote: However I'm only now starting to complain loudly and repeatedly just because I'm seeing such a huge wasted opportunity. Instead of complaining, why don't you put together a more concrete and technical proposal of what you actually want? Because this thread is getting really long with a low fact/opinion ratio. Surely by far not enough facts to convince me. I also see a lot of discussion in this thread about general issues with Sage and Sage development which are unrelated to modularisation. So, when you do write that proposal, be sure to stick to problems which are really caused by the monolithic nature of Sage. Just to be clear: I personally won't be writing any such thing anytime in the near future. I guess my conclusion for now from this thread is not that I'm trying to convince you of anything. However, as a said before, my recommendation to anybody considering writing code for Sage is, to quote Francesco Biscani, to "move out of the reality distortion field and apply normal software development practices." Here's a concrete example of this happening: https://github.com/haochenuw/sage_modabvar Or https://github.com/abelfunctions/abelfunctions https://github.com/coulbois/sage-train-track These packages are nearly impossible to found from the sagemath website! Vincent -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer wrote: > On 2016-04-14 20:35, William Stein wrote: >> >> However I'm only now starting to complain loudly and >> repeatedly just because I'm seeing such a huge wasted opportunity. > > > Instead of complaining, why don't you put together a more concrete and > technical proposal of what you actually want? Because this thread is getting > really long with a low fact/opinion ratio. Surely by far not enough facts to > convince me. > > I also see a lot of discussion in this thread about general issues with Sage > and Sage development which are unrelated to modularisation. So, when you do > write that proposal, be sure to stick to problems which are really caused by > the monolithic nature of Sage. Just to be clear: I personally won't be writing any such thing anytime in the near future. I guess my conclusion for now from this thread is not that I'm trying to convince you of anything. However, as a said before, my recommendation to anybody considering writing code for Sage is, to quote Francesco Biscani, to "move out of the reality distortion field and apply normal software development practices." Here's a concrete example of this happening: https://github.com/haochenuw/sage_modabvar > I can believe that there are certain parts of Sage which could be split up > (you gave the example of 3D plotting and cysignals), but I think that's only > realistic for the non-mathematical parts of Sage. So I wonder if you think > that, say, elliptic curves should be a separate package too? I'm not lowering the fact/opinion ratio further... William -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-14 20:35, William Stein wrote: However I'm only now starting to complain loudly and repeatedly just because I'm seeing such a huge wasted opportunity. Instead of complaining, why don't you put together a more concrete and technical proposal of what you actually want? Because this thread is getting really long with a low fact/opinion ratio. Surely by far not enough facts to convince me. I also see a lot of discussion in this thread about general issues with Sage and Sage development which are unrelated to modularisation. So, when you do write that proposal, be sure to stick to problems which are really caused by the monolithic nature of Sage. I can believe that there are certain parts of Sage which could be split up (you gave the example of 3D plotting and cysignals), but I think that's only realistic for the non-mathematical parts of Sage. So I wonder if you think that, say, elliptic curves should be a separate package too? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Thursday, April 14, 2016, Francesco Biscani wrote: > On 14 April 2016 at 15:51, Erik Bray > wrote: > >> I disagree that Sage is all that special. Or at least, I don't >> believe there's any need for it to be, whether or not it is currently. >> > > If the past is any indication, you will find some cultural resistance > about this point in the Sage community. Sometimes it seems like Sage lives > in a reality distortion field in which normal software development > practices do not apply :) > > William's change of heart about modularization is a relatively recent > development, which - I speculate - might have been brought forward by his > increased involvement with the Python (and non-Python) ecosystems while > working on SMC. > I have had a strong interest in this since when I started psage over 6 years ago. However I'm only now starting to complain loudly and repeatedly just because I'm seeing such a huge wasted opportunity. Let's move out of the reality distortion field and apply normal software development practices!! > > FWIW, I think you are doing the right thing and wish you all the best luck > in the effort :) > > Cheers, > > Francesco. > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-14 17:38, Erik Bray wrote: Sage already has the problem of large chunks of code that are effectively unmaintained and create a maintenance burden on anyone serious about maintaining sage. Their interfaces whither, and become inconsistent with the rest of the package. It's dead weight. I disagree with the above. It's not necessarily a problem to have unmaintained-but-still-working code. Why? I thought the goal was to deprecate sagenb? At long as SageNB is still supposed to work (even if deprecated), somebody has to do some minimal maintenance to keep it working. Currently, it seems that nobody wants to do this. If SageNB would become again part of Sage, then we can do this minimal maintenance as part of normal Sage development. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 5:25 PM, Jeroen Demeyer wrote: > On 2016-04-14 15:46, Erik Bray wrote: >> >> If truly nobody can maintain an affiliated package >> anymore it might die. And that's a problem since it might mean loss of >> functionality for users > > > A logical conclusion from the above that it's a simply a bad idea to split > up Sage into separate packages unless somebody really cares about > maintaining each of those packages. I care about cysignals since I wrote > most of it, but if nobody cared we should not have split it up. I think we're dealing with different premises because that's not the logical conclusion I come to. Sage already has the problem of large chunks of code that are effectively unmaintained and create a maintenance burden on anyone serious about maintaining sage. Their interfaces whither, and become inconsistent with the rest of the package. It's dead weight. cysignals is a different story altogether, since it was split out for a different reason--that it has functionality not particular to sage and not intricately tied to other parts of sage. It's worth splitting out because it can benefit the wider Python community (something most of sage-devel seems to think is on another plane of existence ;) and in turn cysignals (and sage) can benefit from that. matplotlib started out as part of a package of EEG data analysis tools (I think it was always developed as a separate library but I'm not sure--regardless that was the motivation behind it). If it never grew beyond the specific domain that motivated it it would no longer exist. > The converse should also be true: if a package is no longer actively worked > on, it should be folded back into Sage core. I think this would currently be > a good idea for SageNB. Why? I thought the goal was to deprecate sagenb? Easier to do if it's separate--eventually you say "Okay, the last version of sage that sagenb works with is X.Y" then let it whither and die. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-14 15:46, Erik Bray wrote: If truly nobody can maintain an affiliated package anymore it might die. And that's a problem since it might mean loss of functionality for users A logical conclusion from the above that it's a simply a bad idea to split up Sage into separate packages unless somebody really cares about maintaining each of those packages. I care about cysignals since I wrote most of it, but if nobody cared we should not have split it up. The converse should also be true: if a package is no longer actively worked on, it should be folded back into Sage core. I think this would currently be a good idea for SageNB. Jeroen. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 04/14/2016 09:21 AM, Jeroen Demeyer wrote: > A really important point (which so far hasn't been addressed) is how to > deal with breakages of affiliated packages. I see two ways: > > 1. If I make a change to some core package, it is my responsibility to > ensure that no affiliated package breaks. > > 2. It is the responsibility of the affiliated package to fix breakages. > It would be have to be (2), and that would be the distinction between an affiliated package (one that depends on Sage) and one that gets added to Sage itself. Both should still be an option: people get less upset if they make a choice and it bites them in the ass than they would get if you forced the same choice (that they would have made anyway) on them. Breakages will still occur, but they can be managed. If it becomes possible to package Sage without all of third-party software, then it becomes possible for the people who depend on Sage to depend on a particular version (and enforce that with pip, or a package manager, or whatever). That's not possible now because Sage hasn't been packaged most places. Then all we need is a little discipline to not break things too badly between releases. To do that, we have to figure out what the "public API" is, and how much we're allowed to change it. Major versions (e.g. sage-6.x to sage-7.x) should be "anything goes." With the current versioning scheme, minor releases should be things like build-system changes, method additions, bug fixes, and dependency updates. (This needs more careful thought than I've given it.) The release manager would have to decide which is which, and maintain two separate branches. The current "develop" branch corresponds to "the next major release", and we would need a new branch for "the next minor release" containing only those tickets that don't break the public API. Nobody can write code depending on a library that is guaranteed to work forever with all versions of that library. But "anything can change at any time" isn't much of a promise to those people. Having a more semantic versioning scheme will make things a lot less painful. And the option to commit straight into Sage itself is there for the people who are willing to give up some control for stability. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 14 April 2016 at 15:51, Erik Bray wrote: > I disagree that Sage is all that special. Or at least, I don't > believe there's any need for it to be, whether or not it is currently. > If the past is any indication, you will find some cultural resistance about this point in the Sage community. Sometimes it seems like Sage lives in a reality distortion field in which normal software development practices do not apply :) William's change of heart about modularization is a relatively recent development, which - I speculate - might have been brought forward by his increased involvement with the Python (and non-Python) ecosystems while working on SMC. FWIW, I think you are doing the right thing and wish you all the best luck in the effort :) Cheers, Francesco. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 3:26 PM, kcrisman wrote: >> >> I think these are some thoughtful comments, but I also think this is > > > Thanks. > >> >> partly missing the point. This discussion isn't (necessarily) about >> how Sage is packaged and presented for the average user. One can >> certainly put together a metapackage containing all the bells and >> whistles and optional dependencies that one would ever (and never) >> need. > > > I think I more or less already replied to this that I have no philosophical > objection to the proposed project, just some practical concerns, so I won't > repeat myself, especially since it seems that you are among the practical > resources who might be brought to bear on solving those practical concerns. > Which the whole community, I think, would look forward to - *especially* > concerning Windows :) so thanks in advance! Got it. I think they're all valid concerns that need to be addressed. I (or preferably someone else, but I can do it if no one else does) should go back over this thread and summarize the points and counter-points. IMO most of the counter-points are fine to bring up but mostly consist of specific examples of where something went wrong somewhere one time. I think nobody is claiming that any development philosophy will lead to a problem-free process. > PS I just saw Jeroen's most recent update, and that is a good summary of the > practical problem. Sage is not like other software where a breakage (even a > trivial one) just means that package doesn't work; it could affect a lot of > other stuff, especially the "kitchen sink". I disagree that Sage is all that special. Or at least, I don't believe there's any need for it to be, whether or not it is currently. Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 14, 2016 at 3:21 PM, Jeroen Demeyer wrote: > A really important point (which so far hasn't been addressed) is how to deal > with breakages of affiliated packages. I see two ways: These are good questions. > 1. If I make a change to some core package, it is my responsibility to > ensure that no affiliated package breaks. > > 2. It is the responsibility of the affiliated package to fix breakages. Those are the most obvious but there are other ways. A variant on #2 is that it's the responsibility of the affiliated package's maintainer to perform integration testing of their package and report issues that result from upstream changes. But whose responsibility it is to "fix breakages" may depend on the particular issue. I agree with the overall premise though. > I think both of the above have problems: > > With 1, the author of the core package might not have access to the > affiliated package: how do you force your affiliated package to accept a > needed change? This is the current situation with Sage and SageNB. > > With 2, breakages will need to fixed after the fact (after the change to the > core package has been merged already), leading to occasional breakages for > end users. This is the current situation with Sage optional packages. I agree, mostly. Both of these *are* problems. Astropy doesn't do #1--that is, there is nothing that tests Astropy core changes against all affiliated packages (rather, the affiliated packages run their own tests against the latest Astropy core--the inverse). I think there has been some discussion of adding affiliated package tests to Astropy's main contiguous integration matrix and I think that would be a good idea. It would also enable better integration testing between affiliated packages (some already do this--for example APLpy [1] depends on WCSAxes [2]--both are Astropy affiliated packages and use (different) functionality from the Astropy core library. Implementing #1 would seem to require addressing problem #1. However, this needs to be addressed either way--if some changes are made in the core package that break affiliated packages then the affiliated package *may* have to be updated regardless of how the breakage was discovered. If truly nobody can maintain an affiliated package anymore it might die. And that's a problem since it might mean loss of functionality for users, but it's a less bad problem then having unmaintained code sloshing around in the main package (and if a package goes unmaintained it probably didn't have many, if any users beyond its original author anyways). It should be pinned to require the last versions of its dependencies that it did work with so that it can be easily installed in a virtualenv if absolutely needed, and then be sent out to pasture. Problem #2 does result specifically from requiring affiliated packages to do their own integration testing. That's why having more integration testing for the core package would be good too. This is rarely a huge showstopper though. These issues are almost always caught in development and don't affect anyone but the affiliated package author, and usually not for long, as their package will normally still work with the latest stable version. In *some* cases the development versions of an affiliated package will depend on the development version of the core package, but in most of those cases the affiliated package author is working closely with whoever is working on the dependent features in the core package (or often are the same person(s)). Finally, in the rare cases where a release gets out that breaks integration it can be addressed quickly by a patch release. Currently Sage doesn't do patch releases, but it should. Especially if there is a move toward increased modularization. But that's more a separate issue. Erik [1] https://aplpy.github.io/ [2] http://wcsaxes.readthedocs.org/en/latest/ -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> > > I think these are some thoughtful comments, but I also think this is > Thanks. > partly missing the point. This discussion isn't (necessarily) about > how Sage is packaged and presented for the average user. One can > certainly put together a metapackage containing all the bells and > whistles and optional dependencies that one would ever (and never) > need. > I think I more or less already replied to this that I have no philosophical objection to the proposed project, just some practical concerns, so I won't repeat myself, especially since it seems that you are among the practical resources who might be brought to bear on solving those practical concerns. Which the whole community, I think, would look forward to - *especially* concerning Windows :) so thanks in advance! PS I just saw Jeroen's most recent update, and that is a good summary of the practical problem. Sage is not like other software where a breakage (even a trivial one) just means that package doesn't work; it could affect a lot of other stuff, especially the "kitchen sink". -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
A really important point (which so far hasn't been addressed) is how to deal with breakages of affiliated packages. I see two ways: 1. If I make a change to some core package, it is my responsibility to ensure that no affiliated package breaks. 2. It is the responsibility of the affiliated package to fix breakages. I think both of the above have problems: With 1, the author of the core package might not have access to the affiliated package: how do you force your affiliated package to accept a needed change? This is the current situation with Sage and SageNB. With 2, breakages will need to fixed after the fact (after the change to the core package has been merged already), leading to occasional breakages for end users. This is the current situation with Sage optional packages. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 9:58 PM, William Stein wrote: > On Tue, Apr 12, 2016 at 12:50 PM, kcrisman wrote: >> You are right, as I apparently >> didn't make clear, that it would be even better to have people more easily >> able to have separate packages that are quite narrow - and I think that a >> testing framework like R has would probably suffice for this (as long as >> breaking such packages was a blocker for release!). > > It may or may not be, depending on the package and why it broke. > That's up to us to decide. Sorry to keep bringing up Astropy here--it by no means does everything perfectly--but it is a good analog to Sage in many ways due to general similarities between scientific software communities. Anyways one thing that Astropy provides for Affiliated Packages is a "package template" [1] that allows all Affiliated Packages to be built and packaged more or less the same way. They aren't *required* to use the package template, but the vast majority do because it's convenient--it sets them up with the same testing and documentation tools that the Astropy core package uses it. And even more importantly (and the reason I bring this up) is that it includes templates for configuring their package to be built and tested on continuous integration platforms like Travis [2] and AppVeyor [3]. The default templates include Astropy as a build/test dependency, and include separate build job configurations for using both the latest stable release as well as the latest version from git. This means that affiliated package authors typically find out quickly if a change in Astropy breaks their code too. This doesn't happen that often due to the slow deprecation process, but it does happen (probably most commonly if some import changes). We can then work with the maintainers of the individual affiliated packages to resolve the issue--whether it's putting the right astropy-version-specific code in their package, or backing out changes or providing better backward-compatibility in Astropy. Unit testing and continuous integration really make this a mostly painless process, and by providing package authors with the tools to do these things easily we promote best practices. Best, Erik [1] https://github.com/astropy/package-template [2] https://travis-ci.org/ [3] https://www.appveyor.com/ -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 7:33 PM, William Stein wrote: > On Tue, Apr 12, 2016 at 7:11 AM, kcrisman wrote: >> This has been an interesting thread. In the end, I think that some (or a >> lot) of Sage's attractiveness to end users goes away if it becomes a bunch >> of possibly-updated packages that might or might not work with a current >> version of Sage. I always found the "with(plots)" syntax (or whatever it >> was) in Maple very frustrating, and that is presumably a 'core' package; >> having random stuff suddenly not work (let me be clear, because it was left >> behind by Sage core) would be even more so - as has been pointed out several >> times here. > > I am a little annoyed, since this completely misunderstands my proposal. > > 1. My proposal was to make it easier for people to develop new code > independently of the core sage library. This in itself has nothing > to do with taking away from the existing library or in fact changing > it in any way (at least initially). > > 2. Moreover, to directly address your concern, if 3d plotting (say) > were split off as a separate Python package/library, that does *not* > imply that when you download and install Sage, or start it, that you > can even tell the difference. It doesn't mean that the normal visible > public API of Sage changes > at all. Why do you think otherwise? We would just include that > Python library as a standard package, just like the hundred other > standard packages. Volker has mentioned several times how Python > enables doing this sort of thing pretty easily already. Okay, and this is what I get for not reading further downthread before replying--I see we're in agreement here. Erik >> Sage users (and potential ones) I speak with want more than just the "basic" >> functionality, because they want something they can use throughout the >> curriculum and in their own research. There are other (good) tools for >> those who truly won't be doing anything beyond calculus. >> >> Now it's true that some material in Sage probably could have been in >> separate packages, as it's quite specialized - likely a lot of the >> sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves >> are actually more popular, I think). But then there's the opposite problem >> of finding out how to enforce that a package must compile with the most >> recent Sage. This is R's model, but R tends to have a very different type >> of package, one that implements something relatively narrow. Also, we don't >> have the auto-testing resources of R. > > 1. There can and should exist packages that depend on the core Sage > library and have a relatively narrow focus. Why not? > > 2. Maybe we don't have the same auto-testing resources as R does > today. Who is to say we won't in a year or two? > > I'm planning for a future where the Sage project *does* have > resources, and where it is possible to hire 2+ people fulltime to > maintain Sage, like what ODK is doing *right now*. That's what we > should be aiming for. We have it right now (due to ODK) with Erik and > Jereon, at least, and if SageMath Inc succeeds, then it will be > possible to continue and grow this. > > >> >> The problems that Luca and Simon are (rightly) pointing out, in my view, are >> not solved by more modularity - if anything, the problems were because they >> were not part of 'standard' Sage, though I of course am not suggesting they >> should have been part of it. > > Their problems were partly caused by us not supporting and encouraging > the creation of code outside of standard Sage. We should be doing > that 1000x what we are doing now. Right now, we as a community (not > me, but certainly many others) are shockingly discouraging and > negative toward any code that isn't officially in the core sage > library. I think this situation is really baffling to see for a lot > of outsiders. > >> As an example of what happens with the package system, consider several >> Maxima packages (which shall remain unnamed) which don't work well with >> other Maxima commands/flags/packages (no doubt rjf would say we are using >> them improperly, which may be true). Well, they're separate packages, with >> their own maintainers, and I don't know that anyone beyond them takes >> responsibility, and fixing lots and lots of hard-to-track-down bugs once >> you've put in the initial effort is very daunting and time-consuming. So >> they kind of languish, I think - not that some Sage bugs don't too, but >> there is less likelihood that someone else will take the time to work on >> them if it's "just a package", perhaps with its own separate web >> affiliation. > > Just because some people have zero funding and are bad at packaging > doesn't mean that the mainstream standard mature approach to open > source software development, as exemplified in many ecosystems now (R, > Pypi, npm, Debian, etc.) is broken. > >> (I'm sure the same thing applies to many user-contributed Mma and Mapl
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 4:11 PM, kcrisman wrote: > This has been an interesting thread. In the end, I think that some (or a > lot) of Sage's attractiveness to end users goes away if it becomes a bunch > of possibly-updated packages that might or might not work with a current > version of Sage. I always found the "with(plots)" syntax (or whatever it > was) in Maple very frustrating, and that is presumably a 'core' package; > having random stuff suddenly not work (let me be clear, because it was left > behind by Sage core) would be even more so - as has been pointed out several > times here. > > Sage users (and potential ones) I speak with want more than just the "basic" > functionality, because they want something they can use throughout the > curriculum and in their own research. There are other (good) tools for > those who truly won't be doing anything beyond calculus. > > Now it's true that some material in Sage probably could have been in > separate packages, as it's quite specialized - likely a lot of the > sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves > are actually more popular, I think). But then there's the opposite problem > of finding out how to enforce that a package must compile with the most > recent Sage. This is R's model, but R tends to have a very different type > of package, one that implements something relatively narrow. Also, we don't > have the auto-testing resources of R. > > Given that, I'd definitely rather have the kitchen sink, including that > stuff, in Sage - as a *user*, not developer. I know that whatever is in > Sage will stay in it, and it will all be there. Hi, I think these are some thoughtful comments, but I also think this is partly missing the point. This discussion isn't (necessarily) about how Sage is packaged and presented for the average user. One can certainly put together a metapackage containing all the bells and whistles and optional dependencies that one would ever (and never) need. So when bundling a SageMath "product" for users, there's not so much harm in throwing in the kitchen sink (generally, I don't think). I know William doesn't want to cut users off from functionality. I interpreted William's original message as being about how Sage is *developed*, and the current kitchen sink model is I think unsustainable for development. System integration can be hard, but at a certain point it's less hard than developing an overly large single project, IMO. Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Just to take a look at how much different modules deppend on each other right now, I did some quick analysis by looking at the import statements of the .py files in the sage library. The results of this quick experiment are as follows: 1. there are 9 modules in the base of the dependency graph (that is, avery other module imports something from them, or from other modules that imports something on them, or so on). They are schemes, ext, docbuild, version, env, all_cmdline, all, all_notebook, and notebook 2. over them there is a big set of modules that have circular dependencies between them. That is, they import something from the previous set, or from other module in this set. They are probability, data_structures, misc, combinat, server, quadratic_forms, homology, plot, functions, stats, matrix, rings, categories, algebras, calculus, arith, finance, symbolic, interfaces, numerical, modular, repl, groups, databases, libs, parallel, structure, geometry, modules, typeset, graphs, quivers, doctest, sets, gsl, monoids. 3. Then there are 16 modules that import something from the previous two lists, but not from anything on this level. They are: games, tensor, matroids, knots, interacts, crypto, dev, logic, lfunctions, dynamics, sat, coding, game_theory, tests, sandpiles, media 4. Finally, there is manifolds, that imports from tensor and from level (2). As I said, this is just a quick look (for instance, I didn't consider lazy imports). But I think it gives an idea of what is already moreless independent, and what is pretty much entangled with the rest of the components. One would be tempted to say that (2) consists of the "core" of the Sage library, whereas (1) are its dependencies and (3) and (4) would be "extra" modules that deppend on the core library. But again, this is just a first impression after a quick look. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
That sounds like an important point. Definitely creating software and maintaining it are two different tasks that might require different workflows. El miércoles, 13 de abril de 2016, 11:24:16 (UTC+2), Simon King escribió: > > Hi! > > On 2016-04-13, Jeroen Demeyer > > wrote: > > I think the reason that the coding people are moving forward quickly is > > mainly because there is an enthousiastic group just doing stuff. > > Aren't there further examples? IIRC, the matroid code was (or still is?) > developed separately, and then made its way into Sage proper. > > Question to people with computer engineering experience: Shouldn't we > distinguish between *creating* functionality in a modular way (such as: > creating matroid code as an optional package) and *maintaining* it in a > modular way? > > My impression with my group cohomology package seems to suggest to me > that maintaining a separate package is more difficult than creating it > in the first place, and I regret that I didn't start earlier with > binding it more tightly to the Sage code, so that regressions could be > detected and fixed more easily. > > Best regards, > Simon > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Hi! On 2016-04-13, Jeroen Demeyer wrote: > I think the reason that the coding people are moving forward quickly is > mainly because there is an enthousiastic group just doing stuff. Aren't there further examples? IIRC, the matroid code was (or still is?) developed separately, and then made its way into Sage proper. Question to people with computer engineering experience: Shouldn't we distinguish between *creating* functionality in a modular way (such as: creating matroid code as an optional package) and *maintaining* it in a modular way? My impression with my group cohomology package seems to suggest to me that maintaining a separate package is more difficult than creating it in the first place, and I regret that I didn't start earlier with binding it more tightly to the Sage code, so that regressions could be detected and fixed more easily. Best regards, Simon -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
Luca, I read your post but I don't really buy your argument that "rewriting an entire module" is easier with a modular Sage. I think it's equally hard in all cases, modular or not. I think the reason that the coding people are moving forward quickly is mainly because there is an enthousiastic group just doing stuff. If we could get a few people like that together for elliptic curves, maybe do a Sage Days around it, I'm sure we could make as much progress within the current Sage development model. I have a few elliptic-curve related tickets lying around, but just no time to work on them. Jeroen. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-12 21:58, William Stein wrote: If 3d plotting were in a separate python library we would enforce it working in a new release in precisely the same way we do so now with standard packages. Install and test it. Which is very much a pain right now with SageNB. I find it a bad thing that SageNB is a separate project while it's still tested within Sage itself. It's in maintenance mode right now but it breaks doctests now and then if something in Sage changes. That is really annoying because a trivial commit now involves making a new SageNB release. Not fun! I would in fact advocate to merge SageNB back into Sage proper. My conclusion is that we need to treat those packages either as truly separate packages or as truly part of Sage. Anything in between (like SageNB currently) is bad. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> > As for the technical concerns of having a modular library, it seems the > biggest worry is that it will be too hard to orchestrate all of the > inter-dependent components of Sage. At this point, I'm not too concerned on > this front -- many projects in the past have solved these sorts of issues > and I don't think Sage is special in this regard. I would want to see a > more formal technical proposal (something along the lines of a SEP) before > I would feel comfortable criticizing such a proposal. > > That is my feeling exactly right now: I see good arguments for the "going modular" idea, and also good arguments against it. Without a specific proposal of how this idea should be materialized in practice, it is difficult for me to get a position about wheather the pros outweight the cons or not. One idea that has come to my mind by reading this thread is the gentoo metapackages. They are packages that do nothing in reality, but deppend on a bunch of other packages. That way, if you want to install, say, the KDE desktop, you just install the KDE metapackage, and it will pull all the modular parts of KDE as dependencies. So translated to our setting, we could have a Sage "metapackage", that enforces to install all the parts that conform "Sage the software". Maybe we would need a better packaging system than what we have now though. And of course, it remains to see in how many different packages the sage library can be splitted. If there are circular dependencies everywhere, at the end we would end with just one package for the whole sage library. Maybe we should start by taking a look at the dependency graph of the different modules to see exactly where are the possible places for splitting. But again, until we come to a concrete proposal, this is pretty much hand waving. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> >> Well, reading this thread made me think that. Because I don't see how we >> enforce all those "other pieces" working nicely together, so some would >> (perhaps quite quickly) drop by the wayside. You are right, as I >> apparently didn't make clear, that it would be even better to have people >> more easily able to have separate packages that are quite narrow - and I >> think that a testing framework like R has would probably suffice for this >> (as long as breaking such packages was a blocker for release!). (There's >> also the CRAN versus RForge etc. question...) But I don't see how this is >> possible with the *current* Sage. >> >> > Cars are built out of parts. >> >> Yes, parts that are, despite parts suppliers not being the same as the >> car companies, those parts have high tolerances and highly controlled >> supply chains. (Or so my brief experience with a parts supplier long ago >> suggests.) All your examples of "mature open source development" are not >> the same in their goal as Sage, which apparently aims to have all of >> mathematics at its fingertips (in the way that Maple and Mathematica, >> though presumably not Magma or Matlab, seem to). >> > > This also seems to be the crux of why some Sage developers are not > interested in breaking things up (as opposed to why they actively do not > want to break things up) -- namely that Sage should be a monolithic > application just like the Maple and Mathematica, and that is all it should > be. > > While in the early days a lot of Sage's functionality was simply wrapping > other open source libraries, now there is quite a bit that can only be > found in Sage's library -- it seems to me only like the right thing to do > to make it so that others in the open source community could easily benefit > from our work just like we have benefited from theirs. > > This doesn't conflict with our mission statement, as it is still perfectly > possible to create a interactive shell with tons of functionality from a > collection of libraries (again, Sage started as this). > > As for the technical concerns of having a modular library, it seems the > biggest worry is that it will be too hard to orchestrate all of the > inter-dependent components of Sage. At this point, I'm not too concerned on > this front -- many projects in the past have solved these sorts of issues > and I don't think Sage is special in this regard. I would want to see a > more formal technical proposal (something along the lines of a SEP) before > I would feel comfortable criticizing such a proposal. > This is all a good summary, thank you. If someone feels it is worth the energy to do all this, that sounds fine; I would just be concerned that it wouldn't be realistic (it would remain possible, I suppose) to do it without giving up the "monolithic" version for those (which is a lot) who would want it. > Why not in Sympy or elsewhere? As best I can tell it is because of fear like you seem to have... Well, and the nontrivial question of when/where it's going to happen, which will by necessity invoke an opportunity cost with respect to other things that people who know how to do that (a small set) are doing. But it's certainly more a practical objection than a principled one, and in any case I fear to wear out my welcome since I have not been able to actively work on Sage this semester, or year, really. :-( -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Let me share some more thoughts on the interplay between modularization and the review process. I'm afraid I'm going to be long... TL;DR: modularization is good to keep Sage going forward. Our review process discourages patchbombs, and for a reason. For one, no one wants to review patchbombs. But, most importantly, it is too risky to drop a patchbomb into the next release, having the doctests and a couple of human eyes as the only safety guarantee. The consequence of this is that we modify old code by carefully *sprinkling* diffs here and there, breaking up major changes into minor, innocent looking patches. I know something about it, I'm doing it right now for the Pari interface. There is nothing fundamentally bad about this, and, eventually, old code gets fully rewritten and the codebase gets improved. Only, this is time-consuming, and, by definition, it leads to *patchy* code. What I want to highlight is that non-modularization + review process discourage major rewrites too much. Some examples: - How many modules in Sage haven't moved to the category framework yet (assuming that the move would make sense for the module)? - With all the respect I owe to the huge work done by William, John, and the other contributors to elliptic curves in Sage, I've found myself many times thinking "This design is so bad! I want to rewrite it from scratch". On the opposite site, take the sage-coding project: it is making huge leaps forwards, designing non-trivial mathematical interfaces at an impressive pace. How do they manage it? By writing from scratch, at times even by writing outside of Sage. They have no (or, rather, few) pre-existing interfaces to cater for, and they can simply drop small tickets with half-working code because no other code is depending on them. Most of Sage's code was developed in bursts like this one. Only, after some time, you may realize that there is a better design for what you've written some years before, and that's when you're stuck with the problem of redesigning while maintaining old code at the same time. Constantly rewriting from scratch is silly, and that's why the review process is there. But, never rewriting parts of a code that is 10 years old is also silly. By making it impossible to drop patchbombs, we're slowing down progress. If we made Sage more modular, if it was easy to build code *outside* of Sage and then progressively bring it inside Sage once it is mature, then rewriting whole modules would become much easier. Let's say I'm set up on rewriting the elliptic curve modules, and that I've gathered a small team of enthusiasts to help me. Here's my choices right now: - Announce my plan on sage-devel, seek consensus, maybe write a white paper, modify sage.schemes.elliptic_curves little by little with small patches. This would likely keep breaking the public API at each minor release until the rewrite is done. If we keep faith and get to the end, we would end up with *patchy* code. - Announce my plan on sage-devel, seek consensus, start development in a sage.schemes.elliptic_curves.new package, inside-review the small tickets. When the package is ready, go back to sage-devel, propose to replace the old package with the new one, wait a couple of years. Eventually the rewrite will be accepted, old code will break, the only way for users to keep the old interface will be to stop upgrading Sage. Or, most likely, the rewrite will stay in .new, it will be forgotten, and rot there for the rest of its life. Now, suppose that elliptic curves were a *core* Sage module, shipped with Sage, but which you can uninstall nevertheless. Here's how I would go: Start the new elliptic module outside of Sage, play with it, redesign it a few times. When it starts getting serious, write about it on sage-devel, seek consensus, document the public API. When the module gets stable enough, start advertising it, tell people they can try it out with `sage -i defeo/elliptic_curves`. When the module has gained some traction, go back to sage-devel and propose to replace the core package with it. Have more people look at it, reach feature parity with the core package, wait some minor releases. Eventually the replacement gets accepted, a lot of warnings are issued to the users, the new code goes in at the next *major* release, old code can still run (at least for some time) by doing `sage -u elliptic_curves` and `sage -i elliptic_curves_old`, I become the main maintainer of the elliptic curves core package, John is happy because he does not have to do maintenance anymore. Or maybe the replacement is not accepted, then I write an incendiary email on sage-devel saying all Sage devs are fascists, and I'm quitting the community, and tell all my colleagues to `sage -u elliptic_curves` and `sage -i defeo/elliptic_curves`. The community has lost a (bad) developer, but no code was lost in the process :) I understand that some people here are scared of di
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 12:50 PM, kcrisman wrote: > It doesn't mean that the normal visible >> public API of Sage changes >> at all. Why do you think otherwise? >> > > Well, reading this thread made me think that. Because I don't see how we > enforce all those "other pieces" working nicely together, so some would > (perhaps quite quickly) drop by the wayside. You are right, as I > apparently didn't make clear, that it would be even better to have people > more easily able to have separate packages that are quite narrow - and I > think that a testing framework like R has would probably suffice for this > (as long as breaking such packages was a blocker for release!). (There's > also the CRAN versus RForge etc. question...) But I don't see how this is > possible with the *current* Sage. > > > Cars are built out of parts. > > Yes, parts that are, despite parts suppliers not being the same as the car > companies, those parts have high tolerances and highly controlled supply > chains. (Or so my brief experience with a parts supplier long ago > suggests.) All your examples of "mature open source development" are not > the same in their goal as Sage, which apparently aims to have all of > mathematics at its fingertips (in the way that Maple and Mathematica, > though presumably not Magma or Matlab, seem to). > This also seems to be the crux of why some Sage developers are not interested in breaking things up (as opposed to why they actively do not want to break things up) -- namely that Sage should be a monolithic application just like the Maple and Mathematica, and that is all it should be. While in the early days a lot of Sage's functionality was simply wrapping other open source libraries, now there is quite a bit that can only be found in Sage's library -- it seems to me only like the right thing to do to make it so that others in the open source community could easily benefit from our work just like we have benefited from theirs. This doesn't conflict with our mission statement, as it is still perfectly possible to create a interactive shell with tons of functionality from a collection of libraries (again, Sage started as this). As for the technical concerns of having a modular library, it seems the biggest worry is that it will be too hard to orchestrate all of the inter-dependent components of Sage. At this point, I'm not too concerned on this front -- many projects in the past have solved these sorts of issues and I don't think Sage is special in this regard. I would want to see a more formal technical proposal (something along the lines of a SEP) before I would feel comfortable criticizing such a proposal. > That's a lot harder to disentangle than the dozen or more R packages I've > had occasion to use are, and presumably also the case in non-mathematical > software. Modularity in Sage and resolving our import fiasco isn't the > same thing as saying that every folder in Sage has to be a separate > project, which is what you seem to be implying; though since I > misunderstood the very first point, maybe I'm wrong about that too. > > As an example, take game_theory. This is exactly the kind of "narrow" > thing one might think would survive better as a modular bit that could be > imported. Which one of the programs in question it relies on (gambit) is, > in terms of Python, of course. Yet there is a lot of opportunity with > being inside of "real" Sage for them with interconnections with all kinds > of other combinatorial and plotting and other stuff that they would be hard > put to keep current without being in the Sage library. Having access to > all that other math makes the game_theory stuff more powerful. I am > skeptical that math can be pared down to just a few pieces that everything > else fairly loosely holds fast to. > > But like I said, if you or ODK or someone else can make this vision > happen, as long as the end user still gets the kitchen sink without having > to do some wacky import yoga or use possibly-broken packages, I'm not > complaining! "I also don't have a problem with things like psage or any > other such packages" - and if you're right that my view on that is > selection bias, I would be very happy to be wrong. (Well, not really that > happy, because people have worked hard and Sage has had weird upgrades that > broke their stuff. But you know what I mean.) Bring on the automated > testing! :-) > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Andrew -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this gro
Re: [sage-devel] Re: how we develop sage
On 04/12/2016 03:58 PM, William Stein wrote: > > It's all my fault originally, but I think I've learned something about > software engineering during the last few years... and I think we're > doing it wrong. > Don't be too hard on yourself. The fact that every function in Sage is documented, tested, and reviewed is a bigger "pro" than the packaging is a "con." In that respect Sage is a shining beacon in a sea of garbage. The technical problems are relatively easy to solve. Making modularization possible can only be a good thing. We don't have to force a development model on anyone. If people want to add their code to Sage so that it gets doctested in future releases, great. If they want to maintain it as a separate package, great. If Sage becomes so modular that developers can depend on a specific version of sage-the-library, then breakage should be minimized (as long as we don't make crazy changes in minor releases). -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Tuesday, April 12, 2016, Volker Braun wrote: > On Tuesday, April 12, 2016 at 9:59:40 PM UTC+2, William wrote: >> >> We might also be able to make it available >> outside Sage, and it could suddenly be of huge value to the Python >> world. > > > In other words: If we change our mission to "the most useful collection of > libraries" then we should split up the sage library because it would be > very useful to other projects. > Sigh > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tuesday, April 12, 2016 at 9:59:40 PM UTC+2, William wrote: > > We might also be able to make it available > outside Sage, and it could suddenly be of huge value to the Python > world. In other words: If we change our mission to "the most useful collection of libraries" then we should split up the sage library because it would be very useful to other projects. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 12:50 PM, kcrisman wrote: >> It doesn't mean that the normal visible >> public API of Sage changes >> at all. Why do you think otherwise? > > > Well, reading this thread made me think that. Because I don't see how we > enforce all those "other pieces" working nicely together, so some would > (perhaps quite quickly) drop by the wayside. If 3d plotting were in a separate python library we would enforce it working in a new release in precisely the same way we do so now with standard packages. Install and test it.However, by developing it as a separate library, we can clarify how it depends on the rest of sage (and vice versa). We might also be able to make it available outside Sage, and it could suddenly be of huge value to the Python world. In fact, technically the sage 3d plotting has a huge amount of cool highly optimized and useful code for 3d graphics, e.g., the marching cubes implicit plot code, etc., and beautifully written Cython 3d plotting code by Robert Bradshaw. It is a *really stupid* and sad waste that this 3d functionality is *only* available in Sage? Why not in Sympy or elsewhere? As best I can tell it is because of fear like you seem to have... It's all my fault originally, but I think I've learned something about software engineering during the last few years... and I think we're doing it wrong. > You are right, as I apparently > didn't make clear, that it would be even better to have people more easily > able to have separate packages that are quite narrow - and I think that a > testing framework like R has would probably suffice for this (as long as > breaking such packages was a blocker for release!). It may or may not be, depending on the package and why it broke. That's up to us to decide. William -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> > It doesn't mean that the normal visible > public API of Sage changes > at all. Why do you think otherwise? > Well, reading this thread made me think that. Because I don't see how we enforce all those "other pieces" working nicely together, so some would (perhaps quite quickly) drop by the wayside. You are right, as I apparently didn't make clear, that it would be even better to have people more easily able to have separate packages that are quite narrow - and I think that a testing framework like R has would probably suffice for this (as long as breaking such packages was a blocker for release!). (There's also the CRAN versus RForge etc. question...) But I don't see how this is possible with the *current* Sage. > Cars are built out of parts. Yes, parts that are, despite parts suppliers not being the same as the car companies, those parts have high tolerances and highly controlled supply chains. (Or so my brief experience with a parts supplier long ago suggests.) All your examples of "mature open source development" are not the same in their goal as Sage, which apparently aims to have all of mathematics at its fingertips (in the way that Maple and Mathematica, though presumably not Magma or Matlab, seem to). That's a lot harder to disentangle than the dozen or more R packages I've had occasion to use are, and presumably also the case in non-mathematical software. Modularity in Sage and resolving our import fiasco isn't the same thing as saying that every folder in Sage has to be a separate project, which is what you seem to be implying; though since I misunderstood the very first point, maybe I'm wrong about that too. As an example, take game_theory. This is exactly the kind of "narrow" thing one might think would survive better as a modular bit that could be imported. Which one of the programs in question it relies on (gambit) is, in terms of Python, of course. Yet there is a lot of opportunity with being inside of "real" Sage for them with interconnections with all kinds of other combinatorial and plotting and other stuff that they would be hard put to keep current without being in the Sage library. Having access to all that other math makes the game_theory stuff more powerful. I am skeptical that math can be pared down to just a few pieces that everything else fairly loosely holds fast to. But like I said, if you or ODK or someone else can make this vision happen, as long as the end user still gets the kitchen sink without having to do some wacky import yoga or use possibly-broken packages, I'm not complaining! "I also don't have a problem with things like psage or any other such packages" - and if you're right that my view on that is selection bias, I would be very happy to be wrong. (Well, not really that happy, because people have worked hard and Sage has had weird upgrades that broke their stuff. But you know what I mean.) Bring on the automated testing! :-) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 7:11 AM, kcrisman wrote: > This has been an interesting thread. In the end, I think that some (or a > lot) of Sage's attractiveness to end users goes away if it becomes a bunch > of possibly-updated packages that might or might not work with a current > version of Sage. I always found the "with(plots)" syntax (or whatever it > was) in Maple very frustrating, and that is presumably a 'core' package; > having random stuff suddenly not work (let me be clear, because it was left > behind by Sage core) would be even more so - as has been pointed out several > times here. I am a little annoyed, since this completely misunderstands my proposal. 1. My proposal was to make it easier for people to develop new code independently of the core sage library. This in itself has nothing to do with taking away from the existing library or in fact changing it in any way (at least initially). 2. Moreover, to directly address your concern, if 3d plotting (say) were split off as a separate Python package/library, that does *not* imply that when you download and install Sage, or start it, that you can even tell the difference. It doesn't mean that the normal visible public API of Sage changes at all. Why do you think otherwise? We would just include that Python library as a standard package, just like the hundred other standard packages. Volker has mentioned several times how Python enables doing this sort of thing pretty easily already. > Sage users (and potential ones) I speak with want more than just the "basic" > functionality, because they want something they can use throughout the > curriculum and in their own research. There are other (good) tools for > those who truly won't be doing anything beyond calculus. > > Now it's true that some material in Sage probably could have been in > separate packages, as it's quite specialized - likely a lot of the > sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves > are actually more popular, I think). But then there's the opposite problem > of finding out how to enforce that a package must compile with the most > recent Sage. This is R's model, but R tends to have a very different type > of package, one that implements something relatively narrow. Also, we don't > have the auto-testing resources of R. 1. There can and should exist packages that depend on the core Sage library and have a relatively narrow focus. Why not? 2. Maybe we don't have the same auto-testing resources as R does today. Who is to say we won't in a year or two? I'm planning for a future where the Sage project *does* have resources, and where it is possible to hire 2+ people fulltime to maintain Sage, like what ODK is doing *right now*. That's what we should be aiming for. We have it right now (due to ODK) with Erik and Jereon, at least, and if SageMath Inc succeeds, then it will be possible to continue and grow this. > > The problems that Luca and Simon are (rightly) pointing out, in my view, are > not solved by more modularity - if anything, the problems were because they > were not part of 'standard' Sage, though I of course am not suggesting they > should have been part of it. Their problems were partly caused by us not supporting and encouraging the creation of code outside of standard Sage. We should be doing that 1000x what we are doing now. Right now, we as a community (not me, but certainly many others) are shockingly discouraging and negative toward any code that isn't officially in the core sage library. I think this situation is really baffling to see for a lot of outsiders. > As an example of what happens with the package system, consider several > Maxima packages (which shall remain unnamed) which don't work well with > other Maxima commands/flags/packages (no doubt rjf would say we are using > them improperly, which may be true). Well, they're separate packages, with > their own maintainers, and I don't know that anyone beyond them takes > responsibility, and fixing lots and lots of hard-to-track-down bugs once > you've put in the initial effort is very daunting and time-consuming. So > they kind of languish, I think - not that some Sage bugs don't too, but > there is less likelihood that someone else will take the time to work on > them if it's "just a package", perhaps with its own separate web > affiliation. Just because some people have zero funding and are bad at packaging doesn't mean that the mainstream standard mature approach to open source software development, as exemplified in many ecosystems now (R, Pypi, npm, Debian, etc.) is broken. > (I'm sure the same thing applies to many user-contributed Mma and Maple > packages, but I don't listen in on their ecosystems - I mean upgrades > breaking them, that is. They are fairly monolithic, though?) > > To be clear, I'm only talking about the core Sage library; if people can > find a way to make the other stuff more like sage-on-Gentoo without making > i
Re: [sage-devel] Re: how we develop sage
This has been an interesting thread. In the end, I think that some (or a lot) of Sage's attractiveness to end users goes away if it becomes a bunch of possibly-updated packages that might or might not work with a current version of Sage. I always found the "with(plots)" syntax (or whatever it was) in Maple very frustrating, and that is presumably a 'core' package; having random stuff suddenly not work (let me be clear, because it was left behind by Sage core) would be even more so - as has been pointed out several times here. Sage users (and potential ones) I speak with want more than just the "basic" functionality, because they want something they can use throughout the curriculum and in their own research. There are other (good) tools for those who truly won't be doing anything beyond calculus. Now it's true that some material in Sage probably could have been in separate packages, as it's quite specialized - likely a lot of the sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves are actually more popular, I think). But then there's the opposite problem of finding out how to enforce that a package must compile with the most recent Sage. This is R's model, but R tends to have a very different type of package, one that implements something relatively narrow. Also, we don't have the auto-testing resources of R. Given that, I'd definitely rather have the kitchen sink, including that stuff, in Sage - as a *user*, not developer. I know that whatever is in Sage will stay in it, and it will all be there. The problems that Luca and Simon are (rightly) pointing out, in my view, are not solved by more modularity - if anything, the problems were because they were not part of 'standard' Sage, though I of course am not suggesting they should have been part of it. With the kind of auto-testing R has, at least Simon's problems could have been immediately noted and avoided (as a full optional package); Luca's list of breakages is impressive indeed, and a better way of indicating what is "API" and what isn't would have been very useful there, and would be useful now. But you can have a permissive or non permissive set of publicly available stuff with or without lots of separate packages. As an example of what happens with the package system, consider several Maxima packages (which shall remain unnamed) which don't work well with other Maxima commands/flags/packages (no doubt rjf would say we are using them improperly, which may be true). Well, they're separate packages, with their own maintainers, and I don't know that anyone beyond them takes responsibility, and fixing lots and lots of hard-to-track-down bugs once you've put in the initial effort is very daunting and time-consuming. So they kind of languish, I think - not that some Sage bugs don't too, but there is less likelihood that someone else will take the time to work on them if it's "just a package", perhaps with its own separate web affiliation. (I'm sure the same thing applies to many user-contributed Mma and Maple packages, but I don't listen in on their ecosystems - I mean upgrades breaking them, that is. They are fairly monolithic, though?) To be clear, I'm only talking about the core Sage library; if people can find a way to make the other stuff more like sage-on-Gentoo without making it really, really hard to use on any setup other than the most popular distros/most bleeding-edge Mac OS, that is great. I also don't have a problem with things like psage or any other such packages (as I think William is proposing in this thread for much of the functionality), but the experience thus far has been that the successful such things are eventually merged in Sage proper, rather than being contingent. Think of how much time has been spent on managing the sage-combinat queue or branch or whatever to make sure it always works ... *Given our constraints on testing*, building the car was the right idea then, and it's still the right idea now. And I would rather have the car that can drive everywhere - for me, for my students, for my colleagues - against the car that needs fancy upgrades that are often not available on my model. Volker's original comment still holds: "As long as the goal of "import sage" is to give you something like the feature set of Sage right now we don't benefit from modularization. That is just a tautology, the goal is just not a modular one. We'd just shoot ourselves in the foot if we split things into multiple interdependent packages that then must be upgraded in lockstep." -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more optio
Re: [sage-devel] Re: how we develop sage
(1) Sage developers could be better informed of what are they are going to affect if sage, as an "author tool", send usage statistics about each "sage-lib" related function: - Author should put sage in "send usage statistics mode" - The same author would be warned if any of those functions starts to be deprecated. - And maybe, if authors freely send files to be kept and seen as "testing examples" like in "sage -t .sage" But after a new deprecation warning, I'm thinking of implement one of this solutions to my code: (2) A tool to automatically change the deprecated "simplify_exp" for "canonicalize_radical" inside sage files; (3) Create my own local "simplify_exp" that mimics original "simplify_exp" because here authors enjoy applying "simplify_exp" to exponential expressions. Pedro (a developer of a package that uses sagemath and will be in pypi as suggested) sexta-feira, 8 de Abril de 2016 às 17:43:46 UTC+1, William escreveu: > > On Fri, Apr 8, 2016 at 9:33 AM, Luca De Feo > wrote: > >> How often do you feel that changes in Sage breaks your code? > > > > Looking at my log (this project > > https://github.com/defeo/ss-isogeny-software) > > > > - Sage 6.2: module renamed in Pari/GP interface (without deprecation, as > far > > as I recall) > > - Sage 6.4: frankly, I don't know, but possibly a change in the finite > > fields implementation > > - Sage 6.7: changes in cython, not really Sage's fault, I guess > > - Sage 6.7: changes in the finite fields implementation (with proper > > deprecation, this time) > > - Sage 6.10: changes in richcmp (did not get deprecation warnings, but > the > > change concerns an internal function) > > - Sage 7.1: affected by #19941 (properly deprecated, note that 7.1 is > stable > > right now, but not yet in SMC) > > > > It may be argued that I'm taking risks because I sometimes fiddle with > "the > > internals" of Sage, but one other problem of Sage is that it does not > define > > clearly what's the public API and what's internal. > > +1 -- this "one other problem of Sage is that it does not define > clearly what's the public API and what's internal." is a HUGE problem > -- and we're not aware of it because "user code" is typically private. > > If there were a large body of pip-installable packages, which are user > code, this would help *define* what the public API of Sage really is, > and also give us a much larger body of code to test against before > making new releases. > > As an example, when working on > > https://github.com/williamstein/sage_modabvar > > the first thing I noticed in that from sage-6.10 to sage-7.x > >import sage.rings.arith > > completely BROKE. It became "import sage.arith". > > This is (who knows?) a change in the public api of the core sage > library. Maybe it's a gratituous one, and in this case one I'm > puzzled by why this change was made. (I wrote sage.rings.arith in the > first place, and it was meant to be some misc arith needed to > implement rings. Why move it to its own top level module like that?) > > By writing Python packages that depend on "the core sage library" we > will be forced to define what the core library is and stabilize it. > If you guys want Sage to survive/thrive, this **MUST HAPPEN**. The > core sage library should -- like with Python -- be the place where > code, which has been tested/used by people and got stable, goes to > stay stable (at least API-wise). Like the standard Python library. > > William > > > > > Luca > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "sage-devel" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to sage-devel+...@googlegroups.com . > > To post to this group, send email to sage-...@googlegroups.com > . > > Visit this group at https://groups.google.com/group/sage-devel. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > William (http://wstein.org) > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Tue, Apr 12, 2016 at 11:19 AM, Erik Bray wrote: > On Mon, Apr 11, 2016 at 7:36 PM, Volker Braun wrote: >> On Monday, April 11, 2016 at 3:00:12 PM UTC+2, Erik Bray wrote: >>> >>> It's no doubt very time consuming to check every merged pull request >> >> >> Thats a great idea and scales really well when the project merges 5 branches >> a day. It will also ensure that there are always enough volunteers to serve >> as release manager. > > Sorry, I can't tell if this was meant to be sincere or not. Because > I'm being quite sincere. Sage isn't the biggest project in the > world--it can be managed. The problem currently is that it's on an > all-volunteer basis, and that's hard. And I respect that. But I don't > think having an increasingly unmanageable project is going to attract > *more* volunteers. Another reason to look for places to trim the fat, > so to speak. As an aside after reading this over again, I apologize if this came across as not valuing the work you do as release manager. I know it's a time-intensive and often thankless task, especially when it's being done on a volunteer basis. I just wanted to be clear on that. It's certainly *nobody's* fault that sage is the way it is, and not everything about it is bad either (I think the existing issue workflow is great for the most part!). I just think it's a big enough project that it can't afford not to be developed more carefully going forward... Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Mon, Apr 11, 2016 at 8:29 PM, Nils Bruin wrote: > On Monday, April 11, 2016 at 10:46:59 AM UTC-7, Volker Braun wrote: >> >> On Monday, April 11, 2016 at 2:57:16 PM UTC+2, Erik Bray wrote: >>> >>> Sage, unfortunately, hasn't made many pacts in this regard >> >> >> Sage does have a very clear way of making symbols available on the >> commandline, namely via accompanying all.py files. We can either use that to >> define our public api (which it de facto already is when you use Sage >> interactively), or go through every module and sprinkle around underscores. >> Only one of these two is realistic. > > > Are you proposing to define the public API as just the functions that are > available on the command line by default? It sounds attractive, but I think > it's not workable. For instance, in > > sage: k.=GF(5^3) > sage: k.primitive_element() > a > > I think we do have to consider `primitive_element` as part of the public API > (otherwise, how do people get a primitive element in a finite field using > only the public API?) So that would leave the sprinkling around underscores > as the only realistic option ... Would you care to reconsider? This is a perfect example of why class methods are especially critical to have well-defined public/private visibility. There's no __all__-like mechanism for classes, unfortunately (the closest I can think of is __dir__). Thanks, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Mon, Apr 11, 2016 at 7:36 PM, Volker Braun wrote: > On Monday, April 11, 2016 at 3:00:12 PM UTC+2, Erik Bray wrote: >> >> It's no doubt very time consuming to check every merged pull request > > > Thats a great idea and scales really well when the project merges 5 branches > a day. It will also ensure that there are always enough volunteers to serve > as release manager. Sorry, I can't tell if this was meant to be sincere or not. Because I'm being quite sincere. Sage isn't the biggest project in the world--it can be managed. The problem currently is that it's on an all-volunteer basis, and that's hard. And I respect that. But I don't think having an increasingly unmanageable project is going to attract *more* volunteers. Another reason to look for places to trim the fat, so to speak. >> Sage, unfortunately, hasn't made many pacts in this regard > > Sage does have a very clear way of making symbols available on the > commandline, namely via accompanying all.py files. We can either use that to > define our public api (which it de facto already is when you use Sage > interactively), or go through every module and sprinkle around underscores. > Only one of these two is realistic. Why only one? One thing that too many people in this discussion seem to assume is that this has to be all or nothing. You wrote "every module", but why not "some modules", or "the most abused modules"? Or, "on a case-by-case basis when evidence of misuse turns up in bug reports"? Because an incremental approach is better than doing nothing at all. An all.py file is fine too though and certainly helps as a starting point. We could say "don't use anything that's not listed there". But certainly when it comes to class methods, perhaps more than anything else, consistent use of underscores is a must. Module names, and functions within modules are sometimes a little looser, and better handled through __all__ or some equivalent thereof. > Equally important, we need > a mechanism for consumers of the api to avoid fragile dependencies on > private structures. This could be via unit testing, a linter, or an import > meta hook. Not exactly sure what you would use an import hook for in this case, but it sounds like overkill. It sounds instead like a slight cultural shift is needed--that people writing code for sage need to understand what they're writing Python, and should follow the standard best-practices for Python. They should understand what is and isn't for private use, and that they can't rely on private interfaces. Unit testing + linter rules for all code sounds like a good idea though! Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
Excluding accessing sage.* or the module import cache directly There are various language-lawyering ways of expressing this; I appreciate your contribution to find a bulletproof formulating but I think its intuitively clear what is meant. On Monday, April 11, 2016 at 11:11:04 PM UTC+2, vdelecroix wrote: > > including > sage: sage.combinat.permutation.left_action_product([2,1,3,4], [3,1,2]) > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 11/04/16 15:49, Volker Braun wrote: On Monday, April 11, 2016 at 8:29:06 PM UTC+2, Nils Bruin wrote: Are you proposing to define the public API as just the functions that are available on the command line by default? And all the (non-underscored) methods on objects that you can construct starting from globals, of course. Sorry if that wasn't clear. I.e. basically everything you can do in interactive use, excluding manual imports. including sage: sage.combinat.permutation.left_action_product([2,1,3,4], [3,1,2]) Basically all of Sage can be reached from the global namespace without manual import because modules are already loaded... -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Monday, April 11, 2016 at 8:29:06 PM UTC+2, Nils Bruin wrote: > > Are you proposing to define the public API as just the functions that are > available on the command line by default? > And all the (non-underscored) methods on objects that you can construct starting from globals, of course. Sorry if that wasn't clear. I.e. basically everything you can do in interactive use, excluding manual imports. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Monday, April 11, 2016 at 10:46:59 AM UTC-7, Volker Braun wrote: > > On Monday, April 11, 2016 at 2:57:16 PM UTC+2, Erik Bray wrote: >> >> Sage, unfortunately, hasn't made many pacts in this regard > > > Sage does have a very clear way of making symbols available on the > commandline, namely via accompanying all.py files. We can either use that > to define our public api (which it de facto already is when you use Sage > interactively), or go through every module and sprinkle around underscores. > Only one of these two is realistic. > Are you proposing to define the public API as just the functions that are available on the command line by default? It sounds attractive, but I think it's not workable. For instance, in sage: k.=GF(5^3) sage: k.primitive_element() a I think we do have to consider `primitive_element` as part of the public API (otherwise, how do people get a primitive element in a finite field using only the public API?) So that would leave the sprinkling around underscores as the only realistic option ... Would you care to reconsider? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Monday, April 11, 2016 at 2:57:16 PM UTC+2, Erik Bray wrote: > > Sage, unfortunately, hasn't made many pacts in this regard Sage does have a very clear way of making symbols available on the commandline, namely via accompanying all.py files. We can either use that to define our public api (which it de facto already is when you use Sage interactively), or go through every module and sprinkle around underscores. Only one of these two is realistic. This needs to be complemented with automated testing. On the Sage side, so that the public api does not accidentally change. Equally important, we need a mechanism for consumers of the api to avoid fragile dependencies on private structures. This could be via unit testing, a linter, or an import meta hook. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Monday, April 11, 2016 at 3:00:12 PM UTC+2, Erik Bray wrote: > > It's no doubt very time consuming to check every merged pull request Thats a great idea and scales really well when the project merges 5 branches a day. It will also ensure that there are always enough volunteers to serve as release manager. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-11 15:00, Erik Bray wrote: Also some automated checks are put in place which really help. Can you elaborate on this? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Mon, Apr 11, 2016 at 2:56 PM, Jeroen Demeyer wrote: > On 2016-04-11 14:29, Erik Bray wrote: >> >> Don't advertise an interface as stable and then move it around >> between X.Y releases without a deprecation process. Sage-the-library, >> or much of it, is grown up enough (in terms of stability) that it >> should definitely be doing this. > > > Of course we should be doing this, and in many cases we actually do. We have > a 1-year deprecation policy in Sage. Still, it's easy to forget this and > some people might be unaware. > >> Enforcing this should be part of the >> release manger's job (but everyone should help by being vigilant). > > > I cannot speak for the current release manager, but I certainly wouldn't > want to check every merged ticket to check for things which should be > deprecated. I have done this as release manager of Astropy. It's no doubt very time consuming to check every merged pull request, though I do have a checklist for what I check for, and urge other maintainers with merge-to-master rights to go down it as well. Some are better than others about this and I'm more likely to scrutinize some commits more closely than others depending on the committer's past history of checking the policies :) Also some automated checks are put in place which really help. Anyways, it helps to be able to *pay* someone to do this grunt work... :) It's definitely a lot for a volunteer to do. Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Sun, Apr 10, 2016 at 6:29 AM, Jeroen Demeyer wrote: > On 2016-04-09 16:23, William Stein wrote: >> >> Feedback from users makes it clear they need a stable foundation on >> which to build their work > > > And you think having *many* packages with interdependencies which are > independently managed will actually make it more *stable*? Sounds like the > contrary to me... > > I believe you that there is a problem to be fixed here. However, I > absolutely do not understand why modularization of the Sage library would > fix that problem. It absolutely would help. The question, which needs to be considered and discussed seriously, is where the cuts should be made. You're probably right that there are certain parts of Sage that cannot be reasonably cleaved apart in any way that's readily maintainable. But there are other places where cuts are absolutely appropriate. The questions we need to be asking are what are the "core" functionalities of sage-lib, and what are optional features that are built on top of the "core" functionalities. >> My proposal is simple: it would be very valuable to support (both >> culturally and technically) people creating standalone Python packages >> that depend on Sage, developed using current standard open source >> practices, which are at this point in pretty good shape. > > > It's already possible to make Python packages for Sage (it's just Python > packages). Same thing here: I do not see how modularization of the Sage > library would fix this. > > Maybe some small components which are of independent interest (see cysignals > for example) can be split off, but I doubt this is a good idea for the > mathematical part of Sage. The "mathematical" part of sage is pretty broad and needs to be defined. See also my previous message regarding Astropy affiliated-packages. Almost all affiliated-packages constitute "astronomy code", but that doesn't mean they belong in the Astropy "core" package, at least not initially. But having a family of official affiliated packages does a lot mold and constrain development of the core package,. This includes helping to define what constitutes the "stable API" (because in exchange for affiliated packages using the "core" library, the core library is guaranteeing a certain level of stability between versions. Sage, unfortunately, hasn't made many pacts in this regard, so it's hard to know exactly where to begin. I would probably start with the standard convention for all Python which is that anything that is *not* preceded by an underscore is de-facto "public", while anything preceded by an underscore is de-facto "private". If this hasn't been historically well thought out, then oh well. We're stuck with any "public" interfaces being, well, public, and that means if we want to change or get rid of them they have to go through a deprecation process. For sanity's sake some exceptions can be made. For example: 1) If a "private" interface is explicitly known to be used by external code, and is used in a manner which doesn't break things for the internal code, then it can be "upgraded" to a public interface (by explicitly adding a non-underscored version of that interface, and deprecating the formerly private interface). In other words, if a "private" interface was being used by external code, that demonstrates that there may have been a need for the interface to be exposed in the first place (that or somebody was doing something wrong, which if you guarantee that private interfaces are truly private then you're free to tell that person that their code is wrong and that it's only their fault if it breaks :) 2) There may be code--especially entire modules--that do not use underscored names, and yet are not explicitly for public consumption. For example one might have a "foo.bar.utils" module containing utility functions that are only used within the "foo.bar" module, and were never really meant for "public" use. This is fairly common and annoying to change every time it's encountered. So the best thing to do is clearly mark in the "foo.bar.utils" module docstring that it's for internal use only, and make sure to explicitly exclude it from the published API docs so that users won't tend to find it. This is imperfect but better than nothing. 3) Other exceptions can certainly be made on a case-by-case basis. Having a policy for the general case and sticking to it as best as possible, while not requiring a "foolish consistency", is better than having no policy at all. This is not something that can be ignored. Doing so will (and probably already has) lead to a downward spiral of poor design. Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.goog
Re: [sage-devel] Re: how we develop sage
On 2016-04-11 14:29, Erik Bray wrote: Don't advertise an interface as stable and then move it around between X.Y releases without a deprecation process. Sage-the-library, or much of it, is grown up enough (in terms of stability) that it should definitely be doing this. Of course we should be doing this, and in many cases we actually do. We have a 1-year deprecation policy in Sage. Still, it's easy to forget this and some people might be unaware. Enforcing this should be part of the release manger's job (but everyone should help by being vigilant). I cannot speak for the current release manager, but I certainly wouldn't want to check every merged ticket to check for things which should be deprecated. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Fri, Apr 8, 2016 at 1:35 PM, Jeroen Demeyer wrote: > On 2016-04-08 13:26, Luca De Feo wrote: >> >> This is something that should only happen at major version bumps, if ever. > > > That's easily solved: the next stable release of Sage should be Sage 8, then > Sage 9 and so on... > > We do have a 1-year deprecation policy, but of course it's not enforced (and > I wouldn't know how to enforce that). If the author and reviewer of a ticket > don't care about deprecation, then stuff will break. Deprecation really should be enforced by policy. Use a decorator when marking a function or class (or specific feature thereof) deprecated, and include in the decoration the *version* in which it was marked deprecated. When working on prepping releases search for code that has been marked deprecated for a long time and remove it. But definitely not too soon. Don't advertise an interface as stable and then move it around between X.Y releases without a deprecation process. Sage-the-library, or much of it, is grown up enough (in terms of stability) that it should definitely be doing this. Enforcing this should be part of the release manger's job (but everyone should help by being vigilant). Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-09 16:23, William Stein wrote: Feedback from users makes it clear they need a stable foundation on which to build their work And you think having *many* packages with interdependencies which are independently managed will actually make it more *stable*? Sounds like the contrary to me... I believe you that there is a problem to be fixed here. However, I absolutely do not understand why modularization of the Sage library would fix that problem. My proposal is simple: it would be very valuable to support (both culturally and technically) people creating standalone Python packages that depend on Sage, developed using current standard open source practices, which are at this point in pretty good shape. It's already possible to make Python packages for Sage (it's just Python packages). Same thing here: I do not see how modularization of the Sage library would fix this. Maybe some small components which are of independent interest (see cysignals for example) can be split off, but I doubt this is a good idea for the mathematical part of Sage. Jeroen. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Saturday, April 9, 2016 at 3:24:27 PM UTC+1, William wrote:
>
> On Sat, Apr 9, 2016 at 2:49 AM, Dima Pasechnik > wrote:
> >> SMC does inform my frustration with the current limitations on Sage
> >> development.
> >
> > I am afraid we see a conflict of interest here.
> > It is in interests of SMC that Sage is very, very stable... Frozen.
>
> I want Sage to be (massively) easier to develop and use. I want Sage
> usage (and development) to grow beyond its relatively small initial
> user group, where it has been stuck since 2010. I think the best
> way forward is to trust users and developers further, and support them
> with better tools to do development, rather than viewing them as not
> worthy of such (" independent fifedoms a lot of friction").My
>
While better tools are great, in effect you propose to create just this,
independent fifedoms,
however nice your modularisation proposal might sound.
How this (does not) work(s) in practice one can see to an extent with GAP,
or with Python.
I already elaborated upon the former; for the latter, here is a concrete
example,
related to Sage (it broke Sage on OSX at some point),
with a pip-installable package, https://cryptography.io.
They only build with Xcode on OSX (and cannot with gcc), or at least it was
the case, see
https://github.com/pyca/cryptography/issues/1924, which obviously broke
Sage support.
And the response was, well, how an independent fifedom would respond to a
request not
on its agenda, with "well, hmm, we'll fix it perhaps some time in the
future in some way, but
we would not even look at your trivial 3-line patch, thanks."
Yet another (Cython) example was provided by Jeroen in this thread.
> whole goal with Sage was to give all users as much flexibility and
> power as possible, not to centralize authority too much (like with
> Magma).
>
> Feedback from users makes it clear they need a stable foundation on
> which to build their work, and that -- as much as we imagine we are
> providing one -- Sage is not, and things really have got worse (see
> Simon King's message).I base this on the feedback in this thread,
> and the endless feedback I get from users, and so on.
>
> My proposal is simple: it would be very valuable to support (both
> culturally and technically) people creating standalone Python packages
> that depend on Sage, developed using current standard open source
> practices, which are at this point in pretty good shape.
>
> That's it.
>
> I will continue to recommend to everybody I talk with about Sage
> development that we switch from our current massive monolithic
> centralized approach toward standard open source practices, as I
> mentioned at the beginning of this thread.
There is no successful computer algebra project (of the scale of Sage, or,
say, 1/5th of Sage)
that follows what I think you call, for some reason, "standard open source
practices", while effectively meaning
projects like pip and npm (as if Linux kernel is not following "standard OS
practices").
Moreover, Sage modularisation attempts in the past have not led to anything
useful.
I'd rather see you being much more concrete and constructive in your
proposal.
Dima
I hope I have more time
> in the future to make this easier for people. I appreciate all the
> feedback in this thread.
>
> -- William
>
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Sat, Apr 9, 2016 at 2:49 AM, Dima Pasechnik wrote:
>> SMC does inform my frustration with the current limitations on Sage
>> development.
>
> I am afraid we see a conflict of interest here.
> It is in interests of SMC that Sage is very, very stable... Frozen.
I want Sage to be (massively) easier to develop and use. I want Sage
usage (and development) to grow beyond its relatively small initial
user group, where it has been stuck since 2010. I think the best
way forward is to trust users and developers further, and support them
with better tools to do development, rather than viewing them as not
worthy of such (" independent fifedoms a lot of friction").My
whole goal with Sage was to give all users as much flexibility and
power as possible, not to centralize authority too much (like with
Magma).
Feedback from users makes it clear they need a stable foundation on
which to build their work, and that -- as much as we imagine we are
providing one -- Sage is not, and things really have got worse (see
Simon King's message).I base this on the feedback in this thread,
and the endless feedback I get from users, and so on.
My proposal is simple: it would be very valuable to support (both
culturally and technically) people creating standalone Python packages
that depend on Sage, developed using current standard open source
practices, which are at this point in pretty good shape.
That's it.
I will continue to recommend to everybody I talk with about Sage
development that we switch from our current massive monolithic
centralized approach toward standard open source practices, as I
mentioned at the beginning of this thread. I hope I have more time
in the future to make this easier for people. I appreciate all the
feedback in this thread.
-- William
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Saturday, April 9, 2016 at 12:17:21 AM UTC+1, William wrote: > > > > On Friday, April 8, 2016, Dima Pasechnik > > wrote: > >> >> >> On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote: >>> >>> >>> >>> On Friday, April 8, 2016, Volker Braun wrote: >>> On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote: > > this "one other problem of Sage is that it does not define > clearly what's the public API and what's internal. IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages. >>> >>> >>> Why does nobody ever ask? >>> >>> If there were large body of pip-installable packages, which are user > code, this would help *define* what the public API of Sage really is, > and also give us a much larger body of code to test against before > making new releases. > >> testing against sufficiently large body of code which is not maintained >> by a project is a perfect way to make >> sure that no new releases are made by the project, ever. >> >> >> >> >> >> > > False. You are simply arguing for ignorance. How we chose to use the > results of such testing is up to us to decide. > No, I am arguing against splitting Sage into independent fifedoms. And I have very good reasons for this. Hell, even a model where the kernel is relatively small, and there is a largish collection of independent packages (cf. GAP), functions with a lot of friction between core devs and the rest, with some package authors often endlessly pedalling back kernel changes, kernel development happening in a fork, etc etc is not something one would look forward too. A large part of work for the release manager is being a politician, not a developer, etc etc. And the reason for this is trivial - many package authors don't like to touch their code, once it is written. Which is understandable, but counter-productive for kernel development. > > >> >> >>> What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? >>> I don't care about design schools. I would much rather be aware of how >>> sage-dependent code is actually being used in the wild than to sit in >>> school blissfully ignorant of how sage is really being used. >>> >> >> I thought that with SMC you have a near-perfect opportunity to see what >> Sage users use in the wild... >> > > SMC does inform my frustration with the current limitations on Sage > development. > I am afraid we see a conflict of interest here. It is in interests of SMC that Sage is very, very stable... Frozen. > > > >> >> And perhaps, perhaps, the 1st thing would be to get a single-user SMC >> frontend available as a pip-installable package, so that sagenb can retire, >> at last? >> > > SMC is not a Python program. > and because of this, it cannot get python bindings? Dima > > William > > > >> >> Dima >> >> >>> >>> >>> > > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> Sent from my massive iPhone 6 plus. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-devel+unsubscr...@googlegroups.com. >> To post to this group, send email to sage-devel@googlegroups.com. >> Visit this group at https://groups.google.com/group/sage-devel. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Sent from my massive iPhone 6 plus. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016, Dima Pasechnik wrote: > > > On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote: >> >> >> >> On Friday, April 8, 2016, Volker Braun wrote: >> >>> On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote: this "one other problem of Sage is that it does not define clearly what's the public API and what's internal. >>> >>> >>> IMHO thats just not true; What you get on the commandline (i.e. from >>> sage.all import *) is public and the rest is not. If thats not enough (and >>> really nobody ever asked) we could mark extra imports as public, e.g. by >>> adding special sage.foo.public packages. >>> >> >> >> Why does nobody ever ask? >> >> >>> >>> If there were large body of pip-installable packages, which are user code, this would help *define* what the public API of Sage really is, and also give us a much larger body of code to test against before making new releases. >>> > testing against sufficiently large body of code which is not maintained by > a project is a perfect way to make > sure that no new releases are made by the project, ever. > > > > > > False. You are simply arguing for ignorance. How we chose to use the results of such testing is up to us to decide. > > >> >>> What API design school is that? You dump code on users and whoever >>> manages to build the most convoluted contraption out of that will determine >>> the future direction of the project ;-) Where is the leadership there? Who >>> is going to handle the testing for each ticket, are you going to do that >>> yourself? >>> >>> >>> >> I don't care about design schools. I would much rather be aware of how >> sage-dependent code is actually being used in the wild than to sit in >> school blissfully ignorant of how sage is really being used. >> > > I thought that with SMC you have a near-perfect opportunity to see what > Sage users use in the wild... > SMC does inform my frustration with the current limitations on Sage development. > > And perhaps, perhaps, the 1st thing would be to get a single-user SMC > frontend available as a pip-installable package, so that sagenb can retire, > at last? > SMC is not a Python program. William > > Dima > > >> >> >> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sage-devel" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sage-devel+unsubscr...@googlegroups.com. >>> To post to this group, send email to sage-devel@googlegroups.com. >>> Visit this group at https://groups.google.com/group/sage-devel. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> Sent from my massive iPhone 6 plus. >> > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 8:40:02 PM UTC+2, William wrote: > > Why does nobody ever ask? > I'd like to think thats because there is simply no need for obscure imports when using Sage; We have public constructors for basic mathematical objects, and computations are then done using methods of said objects. Unless you muck in the Sage internals (=you want to patch Sage) you *shouldn't* have to import stuff thats not already global on the commandline, how else are users supposed to use it? What API design school is that? You dump code on users and whoever manages >> to build the most convoluted contraption out of that will determine the >> future direction of the project >> > I don't care about design schools > I disagree, IMHO designing interfaces carefully is much more important than (and should come before) banging on the keyboard. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote: > > > > On Friday, April 8, 2016, Volker Braun > > wrote: > >> On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote: >>> >>> this "one other problem of Sage is that it does not define >>> clearly what's the public API and what's internal. >> >> >> IMHO thats just not true; What you get on the commandline (i.e. from >> sage.all import *) is public and the rest is not. If thats not enough (and >> really nobody ever asked) we could mark extra imports as public, e.g. by >> adding special sage.foo.public packages. >> > > > Why does nobody ever ask? > > >> >> If there were large body of pip-installable packages, which are user >>> code, this would help *define* what the public API of Sage really is, >>> and also give us a much larger body of code to test against before >>> making new releases. >>> >> testing against sufficiently large body of code which is not maintained by a project is a perfect way to make sure that no new releases are made by the project, ever. > >> What API design school is that? You dump code on users and whoever >> manages to build the most convoluted contraption out of that will determine >> the future direction of the project ;-) Where is the leadership there? Who >> is going to handle the testing for each ticket, are you going to do that >> yourself? >> >> >> > I don't care about design schools. I would much rather be aware of how > sage-dependent code is actually being used in the wild than to sit in > school blissfully ignorant of how sage is really being used. > I thought that with SMC you have a near-perfect opportunity to see what Sage users use in the wild... And perhaps, perhaps, the 1st thing would be to get a single-user SMC frontend available as a pip-installable package, so that sagenb can retire, at last? Dima > > > >> >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-devel+unsubscr...@googlegroups.com. >> To post to this group, send email to sage-devel@googlegroups.com. >> Visit this group at https://groups.google.com/group/sage-devel. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Sent from my massive iPhone 6 plus. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016, Volker Braun wrote: > On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote: >> >> this "one other problem of Sage is that it does not define >> clearly what's the public API and what's internal. > > > IMHO thats just not true; What you get on the commandline (i.e. from > sage.all import *) is public and the rest is not. If thats not enough (and > really nobody ever asked) we could mark extra imports as public, e.g. by > adding special sage.foo.public packages. > Why does nobody ever ask? > > If there were large body of pip-installable packages, which are user >> code, this would help *define* what the public API of Sage really is, >> and also give us a much larger body of code to test against before >> making new releases. >> > > What API design school is that? You dump code on users and whoever manages > to build the most convoluted contraption out of that will determine the > future direction of the project ;-) Where is the leadership there? Who is > going to handle the testing for each ticket, are you going to do that > yourself? > > > I don't care about design schools. I would much rather be aware of how sage-dependent code is actually being used in the wild than to sit in school blissfully ignorant of how sage is really being used. > >> >> >> -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote: > > this "one other problem of Sage is that it does not define > clearly what's the public API and what's internal. IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages. If there were large body of pip-installable packages, which are user > code, this would help *define* what the public API of Sage really is, > and also give us a much larger body of code to test against before > making new releases. > What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 6:50:19 PM UTC+2, William wrote:
>
> > But then there is nothing to do on the Sage side, this already works and
> is
> > totally standard.
Just because
> it is technically possible to do something and documented how to do so
> online, doesn't mean there is nothing to do on the sage side.
>
What I meant was: there is nothing that we need to implement to make it
work. As I also mentioned, it would surely be good to have examples and
documentation.
> > Much of the memory usage of the current docbuild is the huge intersphinx
> > data, and no amount of modularization would make that smaller.
> I disagree. If the elliptic curves docs aren't in Sage at all (say),
> then it certainly
> isn't going to make that stuff larger.
>
But not smaller either, unless you get rid of the intersphinx data. The
reference manual is already broken up into smaller chunks that are
processed independently. Breaking it up more is never going to give us a
significant improvement. We can only make sphinx more efficient or build
shittier docs. And there you are IMHO just throwing out the baby with the
bathwater. Remove (or making inaccessible) large chunks of our
documentation and/or disabling hyperlinks surely saves a lot of time and
ram when building the documentation. But why? If you don't care about the
documentation then just compile Sage without it in the first place ("make
build").
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-08 18:43, William Stein wrote: the first thing I noticed in that from sage-6.10 to sage-7.x import sage.rings.arith completely BROKE. That's totally exaggerated. It didn't completely BREAK, it still works with a nice deprecation warning: Importing is_prime from here is deprecated. If you need to use it, please import it directly from sage.arith.all See http://trac.sagemath.org/19879 for details. Saying that stuff BREAKS when people take the effort to make proper deprecation is not encouraging those people to care about deprecations. (disclaimer: I made that change) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Fri, Apr 8, 2016 at 1:00 AM, Volker Braun wrote: > On Friday, April 8, 2016 at 2:36:23 AM UTC+2, William wrote: >>> >>> Whats wrong with the obvious solution: make it a Python package >>> (basically add setup.py) and then "sage -pip install >>> https://github.com/vbraun/awesomepackage.git";. Clearly we could have more >>> documentation for how to write the setup.py or some package directory >>> service. But really it is already a one-liner. >> >> >> That is exactly what I'm proposing > > > But then there is nothing to do on the Sage side, this already works and is > totally standard. Documentation for how to package your own Python code can > easily be found online. (Volker, I really hate to disagree with you, since I respect you *very* highly.) However, this is where you are wrong. Just because it is technically possible to do something and documented how to do so online, doesn't mean there is nothing to do on the sage side. Software, Sage, etc. is all about people, what they endorse, what examples they set, etc. It's a leadership thing. The very thing that makes you say "there is nothing to do" is what tells me "there is a lot to do". > > >>> >>> Yes but cython and cysignals don't do anything for math directly, they >>> are pure dependencies. By contrast, lets say we break out all elliptic curve >>> stuff into a separate package. Just like with cysignals, this will remove >>> all elliptic curve documentation from the Sage reference manual. Still a >>> win-win? >> >> 1. Not necessarily true. > > > It is if you want to have a uniform document with cross references working. I continue to disagree. > Much of the memory usage of the current docbuild is the huge intersphinx > data, and no amount of modularization would make that smaller. I disagree. If the elliptic curves docs aren't in Sage at all (say), then it certainly isn't going to make that stuff larger. > That is not > to say that docbuilding can't be improved, but certainly not by spreading > out the documentation sources. > >> >> 2. Is it so bad that the Cython docs aren't in the sage reference manual? > > > As I said before, cython and cysignals don't do anything for math directly > so they are a special case. In fact, they shouldn't be in the main Sage docs > even if they were part of Sage. And for 99.99% of sage users who don't know an elliptic curve from an "elliptical curve" there is no gain by having the elliptic curves in the same daunting reference manual as plotting functions. > > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
> I merely said that this appears to work. I never said that everything should go there. Sorry for misunderstanding you. > On the other hand, you mention a core Magma library that helps it beat Sage on benchmarks, I imagine such a library would be a nice fit into core Sage. It would be lovely, but unfortunately the licence forbids us to distribute it :( But we're drifting away from the subject. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Fri, Apr 8, 2016 at 9:33 AM, Luca De Feo wrote: >> How often do you feel that changes in Sage breaks your code? > > Looking at my log (this project > https://github.com/defeo/ss-isogeny-software) > > - Sage 6.2: module renamed in Pari/GP interface (without deprecation, as far > as I recall) > - Sage 6.4: frankly, I don't know, but possibly a change in the finite > fields implementation > - Sage 6.7: changes in cython, not really Sage's fault, I guess > - Sage 6.7: changes in the finite fields implementation (with proper > deprecation, this time) > - Sage 6.10: changes in richcmp (did not get deprecation warnings, but the > change concerns an internal function) > - Sage 7.1: affected by #19941 (properly deprecated, note that 7.1 is stable > right now, but not yet in SMC) > > It may be argued that I'm taking risks because I sometimes fiddle with "the > internals" of Sage, but one other problem of Sage is that it does not define > clearly what's the public API and what's internal. +1 -- this "one other problem of Sage is that it does not define clearly what's the public API and what's internal." is a HUGE problem -- and we're not aware of it because "user code" is typically private. If there were a large body of pip-installable packages, which are user code, this would help *define* what the public API of Sage really is, and also give us a much larger body of code to test against before making new releases. As an example, when working on https://github.com/williamstein/sage_modabvar the first thing I noticed in that from sage-6.10 to sage-7.x import sage.rings.arith completely BROKE. It became "import sage.arith". This is (who knows?) a change in the public api of the core sage library. Maybe it's a gratituous one, and in this case one I'm puzzled by why this change was made. (I wrote sage.rings.arith in the first place, and it was meant to be some misc arith needed to implement rings. Why move it to its own top level module like that?) By writing Python packages that depend on "the core sage library" we will be forced to define what the core library is and stabilize it. If you guys want Sage to survive/thrive, this **MUST HAPPEN**. The core sage library should -- like with Python -- be the place where code, which has been tested/used by people and got stable, goes to stay stable (at least API-wise). Like the standard Python library. William > > Luca > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
> How often do you feel that changes in Sage breaks your code? Looking at my log (this project https://github.com/defeo/ss-isogeny-software) - Sage 6.2: module renamed in Pari/GP interface (without deprecation, as far as I recall) - Sage 6.4: frankly, I don't know, but possibly a change in the finite fields implementation - Sage 6.7: changes in cython, not really Sage's fault, I guess - Sage 6.7: changes in the finite fields implementation (with proper deprecation, this time) - Sage 6.10: changes in richcmp (did not get deprecation warnings, but the change concerns an internal function) - Sage 7.1: affected by #19941 (properly deprecated, note that 7.1 is stable right now, but not yet in SMC) It may be argued that I'm taking risks because I sometimes fiddle with "the internals" of Sage, but one other problem of Sage is that it does not define clearly what's the public API and what's internal. Luca -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 1:45:00 PM UTC+1, Luca De Feo wrote: > > > if research developers 1) don't care enough about their own code to > polish it enough for going through our review process > > I don't agree with Dima that putting their code into Sage is what > researchers should do, no matter what. > I merely said that this appears to work. I never said that everything should go there. On the other hand, you mention a core Magma library that helps it beat Sage on benchmarks, I imagine such a library would be a nice fit into core Sage. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Hi Johan, On 2016-04-08, Johan S R Nielsen wrote: > How often do you feel that changes in Sage breaks your code? As I stated before, several times in one year (that year being 2013 or 2014, I am not sure), after it was more or less stable since 2009. Cheers, Simon -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> I don't agree with Dima that putting their code into Sage is what > researchers should do, no matter what. > > I do care a lot about my research code, and I know some of it has no place > inside Sage. Maintaining it outside of Sage is not just carelessness or > laziness, it's a design choice. For sure - and you were not in the group of people I meant, since you clearly stated you cared enough to rebase your code. Changes to Sage so that external packages like yours break is unfortunate. But I'm questioning how much we should shape our development methodology around research projects whose developers *don't* care enough to fix their code when it breaks. And it will break once in a while, as long as it is not part of Sage proper. How often do you feel that changes in Sage breaks your code? Best, Johan -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Hi Dima, On 2016-04-08, Dima Pasechnik wrote: > mind you, Magma group invites such people for paid months-long visits, > where they work on implementing these algorithms in close contact with > Magma core devs. > If only Sage could do this too... :-) Sage has SageDays, and Sage has a rather responsive community. I recall that in the past, the only serious issue in cooperative projects in Sage was the time difference between Europe/Africa, Americas and Asia. But that could be solved by staying up late. So, I think it *is* possible to implement algorithms in close contact with Sage core devs. > or it might make things worse, as the functionality becomes a function of a > multitude of versions of different parts... +1 And *making* it modular in the first place or even just *coping* with modularisation is something that doesn't lie within the core competences of (most) Sage developers, IMHO. > But there is a way to make code work in Sage as upgrades happen - making it > a part of Sage :-) Indeed that's the approach that I now take with my optional group cohomology spkg. Thereby making it modular, just to give that concept a try. Splitting off one third-party component, upgrading to the current upstream version; rebasing a second third party component on top of the upgraded first third party component (that's the main obstacle); and eventually rebasing my own code on top of it, putting it into the Sage library rather than keeping it in one spkg together with the other two components. But it costs me resources that I'd prefer to spend in a more productive way --- and *nothing* came out of it yet! The attempt to modularise and upgrade one upstream source to cooperate with a more recent version of another upstream source takes longer than creating the original version of the spkg! > Thus I don't buy this "future-proof" statement +1 > (well, of course, Sage may > fold if sufficient number of right people get hit by a bus - but the same > is true for Magma) Perhaps that's the hidden purpose of SageDays. Let's make a conspiracy theory out of it ;-) Cheers, Simon -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
> if research developers 1) don't care enough about their own code to polish it enough for going through our review process I don't agree with Dima that putting their code into Sage is what researchers should do, no matter what. I do care a lot about my research code, and I know some of it has no place inside Sage. Maintaining it outside of Sage is not just carelessness or laziness, it's a design choice. Luca -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Wed, Apr 6, 2016 at 9:51 AM, Volker Braun wrote: > On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote: >> >> [...] toward standard open source practices. > > > You mean like in the Linux kernel, which uses a single monolithic git > repository? No piece of user-mode software is so special that it should look to the Linux kernel as a good project to emulate. > Really, modularization is not a useful goal in of itself. And it comes with > its own sets of issues, see the left-pad fiasco last week when the nodejs > clown boat caught fire. That kind of issue is not so common though which is why it was such a fiasco. If this happened all the time you wouldn't even hear about it. [As an aside I wouldn't characterize an entire developer community as a 'clown boat'] -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
> But there is a way to make code work in Sage as upgrades happen - making it > a part of Sage :-) > Thus I don't buy this "future-proof" statement (well, of course, Sage may > fold if sufficient number of right people get hit by a bus - but the same > is true for Magma) It has something to do with the friction that William mentioned, though: researchers barely has the time and motivation to implement their algorithms, but they don't want to afterwards go through the - for new-comers - rather arduous trac/ticket/reviewing process. One reason is probably that they don't want to mature the doc and API of their code sufficiently for it to enter sage-proper. I used to maintain a coding theory library on bitbucket for exactly this reason. Their work could be more available if Sage wasn't so monolithic, or at least, if there was more of a culture of having small Sage packages floating around. If you want their not-completely-nicely-documented code, install their package into your Sage and be at your own risk. What Luca is talking about poses a problem for that strategy though: their code will work with only a few versions of Sage before breaking. Note that I don't personally share this experience: in my own coding theory library for Sage, no function was (AFAIK) broken in the last 5 years due to a Sage update (except for my own bugs, and some stuff related to non-standard packaging). But of course: if research developers 1) don't care enough about their own code to polish it enough for going through our review process, and 2) don't care enough to fix it when incompatibilities with Sage occurs; then is it really code we should be caring about? Won't it quickly rot on github anyway? The research code going into Magma is exactly polished during those visits to Magma HQ that Dima mentions. Best, Johan -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Hi Jeroen, On 2016-04-08, Jeroen Demeyer wrote: > On 2016-04-08 13:26, Luca De Feo wrote: >> This is something that should only happen at major version bumps, if ever. > > That's easily solved: the next stable release of Sage should be Sage 8, > then Sage 9 and so on... > > We do have a 1-year deprecation policy, but of course it's not enforced > (and I wouldn't know how to enforce that). If the author and reviewer of > a ticket don't care about deprecation, then stuff will break. Or rather if the release manager believes that a change is "internal" and doesn't need deprecation, then stuff will break. Cheers, Simon -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Hi Luca, On 2016-04-08, Luca De Feo wrote: > Let me recall something that comes up often, but that I haven't seen yet in > this thread nor at the Sage days. Just two weeks ago I was said: "We are > going to make our student write code in Magma, because we want something > future proof. We've used Sage a couple of times before, but each time Sage > updates break our code in <1yr, and it takes too much work to know why and > update our code". +1 I was seriously fed up when my research software was broken by backwards incompatible ostensibly "internal" changes --- repeatedly within one year---, which hempered my work on a funded project to the extent that I couldn't successfully finish. I considered quitting Sage for that reason. Mainly I didn't quit because in the *past* Sage used to be an ideal platform for my purposes, and while it was moving forward it didn't break too much of existing code --- thus, maintanance of my package and persistence of data was much less of a hastle than it has been in the past one or two years. Since it was OK till perhaps two years ago, I still hope Sage will be able to revive its strength. > I have this experience myself, though, being more involved in Sage, I make > the effort to patch my code to work with the latest stable. At some points > in time, it has even been impossible to have code that works both with the > latest stable and the latest beta (and/or the version in SMC). This is > something that should only happen at major version bumps, if ever. +1 > Of course, this is not easy to solve, but having a more modular > distribution (at least for some meaning of "modular") might help. -1. My impression was that what I perceive as "decay" of Sage was caused by changing Sage's development model. Changing the whole architecture would create even more of an overhead, or rather a swamp in which Sage would disappear. Sage should build the car and not re-invent the wheel --- and much less Sage should re-invent Sage. Best regards, Simon -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 12:26:22 PM UTC+1, Luca De Feo wrote: > > > This thread is first and foremost about reducing the friction involved > in making code that depends on the Sage distribution available to the world. > > Based on feedback I get from users, this friction is a very, very > significant problem to the growth of Sage. > > Let me recall something that comes up often, but that I haven't seen yet > in this thread nor at the Sage days. Just two weeks ago I was said: "We are > going to make our student write code in Magma, because we want something > future proof. We've used Sage a couple of times before, but each time Sage > updates break our code in <1yr, and it takes too much work to know why and > update our code". > > Let me stress that these are highly skilled researchers who develop > amazing software and have a very deep knowledge about programming. One of > them develops a library that is a core piece of infrastructure for Magma, > and which is responsible for a good fraction of the benchmarks where Magma > crushes Sage. If we could make these people stick to Sage, I can only > imagine benefits for us. > mind you, Magma group invites such people for paid months-long visits, where they work on implementing these algorithms in close contact with Magma core devs. If only Sage could do this too... :-) > > I have this experience myself, though, being more involved in Sage, I make > the effort to patch my code to work with the latest stable. At some points > in time, it has even been impossible to have code that works both with the > latest stable and the latest beta (and/or the version in SMC). This is > something that should only happen at major version bumps, if ever. > > Of course, this is not easy to solve, but having a more modular > distribution (at least for some meaning of "modular") might help. > or it might make things worse, as the functionality becomes a function of a multitude of versions of different parts... But there is a way to make code work in Sage as upgrades happen - making it a part of Sage :-) Thus I don't buy this "future-proof" statement (well, of course, Sage may fold if sufficient number of right people get hit by a bus - but the same is true for Magma) > > -- > Luca > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-08 13:26, Luca De Feo wrote: This is something that should only happen at major version bumps, if ever. That's easily solved: the next stable release of Sage should be Sage 8, then Sage 9 and so on... We do have a 1-year deprecation policy, but of course it's not enforced (and I wouldn't know how to enforce that). If the author and reviewer of a ticket don't care about deprecation, then stuff will break. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
> This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world. > Based on feedback I get from users, this friction is a very, very significant problem to the growth of Sage. Let me recall something that comes up often, but that I haven't seen yet in this thread nor at the Sage days. Just two weeks ago I was said: "We are going to make our student write code in Magma, because we want something future proof. We've used Sage a couple of times before, but each time Sage updates break our code in <1yr, and it takes too much work to know why and update our code". Let me stress that these are highly skilled researchers who develop amazing software and have a very deep knowledge about programming. One of them develops a library that is a core piece of infrastructure for Magma, and which is responsible for a good fraction of the benchmarks where Magma crushes Sage. If we could make these people stick to Sage, I can only imagine benefits for us. I have this experience myself, though, being more involved in Sage, I make the effort to patch my code to work with the latest stable. At some points in time, it has even been impossible to have code that works both with the latest stable and the latest beta (and/or the version in SMC). This is something that should only happen at major version bumps, if ever. Of course, this is not easy to solve, but having a more modular distribution (at least for some meaning of "modular") might help. -- Luca -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote: > > Hi, > > This was a comment I just put on trac #965: "I would make a completely > separate python package, maybe called pysmalljac, which builds > smalljac and makes it usable from Python. It would be on github and > pypi. That's how most Sage development should be done. What a > monster I've created by following the Magma way of doing things > instead of the standard open source best practices..." > > I am recommending to absolutely everybody I talk with about Sage > development that we switch from our current massive monolithic > centralized approach toward standard open source practices. Namely, > lots of smaller libraries, standard open source practices, etc. It > would be really valuable to have a thread on sage-devel about how to > more systematically support this. > > Some thoughts: > > - For now, work can be done that is valuable but doesn't have to > impact the current sage/trac workflow. For example, somebody might > create an awesome Python package that does BLAH and depends on the > core Sage library (what you get via "import sage" now). > > - There are 77989 examples of Python packages at > https://pypi.python.org/pypi > > - This is pretty useful to read: > https://python-packaging.readthedocs.org/en/latest/ Jeroen suggests also this link which seems to him more up to date: https://packaging.python.org/en/latest/distributing/ > > > - In the beginning I put everything into one library, since it was > small. Then it started getting bigger and in 2006, 2007, etc. we > talked a lot about splitting up Sage, having smaller packages, etc. > There was a lot of momentum and it was pretty clear to all the CS-type > people involved in Sage (who knew about programming from more than > just Sage) that this is what we needed to do. People like Martin > Albrecht and David Harvey. Then those people all got jobs and > couldn't work on infrastructure aspects of Sage... and nothing > happened in this direction. > >- PSage was an attempt at something like this where I could work > out how to build Cython code in parallel, etc., etc. using more modern > packaging. That was in about 2011, and then I stopped to work on SMC > fulltime for several years, hence work on psage stopped, since I can > only do one thing at a time. > > -- William > > -- > William (http://wstein.org) > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On 2016-04-08 00:58, William Stein wrote: - Cython: relying on pxi/pxd files from the sage library. Please push for https://github.com/cython/cython/pull/483 if you care about this. Cython upstream doesn't seem to "get" that we need this patch. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 2:36:23 AM UTC+2, William wrote: > > Whats wrong with the obvious solution: make it a Python package (basically >> add setup.py) and then "sage -pip install >> https://github.com/vbraun/awesomepackage.git";. Clearly we could have >> more documentation for how to write the setup.py or some package directory >> service. But really it is already a one-liner. >> > > That is exactly what I'm proposing > But then there is nothing to do on the Sage side, this already works and is totally standard. Documentation for how to package your own Python code can easily be found online. > Yes but cython and cysignals don't do anything for math directly, they are >> pure dependencies. By contrast, lets say we break out all elliptic curve >> stuff into a separate package. Just like with cysignals, this will remove >> all elliptic curve documentation from the Sage reference manual. Still a >> win-win? >> > 1. Not necessarily true. > It is if you want to have a uniform document with cross references working. Much of the memory usage of the current docbuild is the huge intersphinx data, and no amount of modularization would make that smaller. That is not to say that docbuilding can't be improved, but certainly not by spreading out the documentation sources. > 2. Is it so bad that the Cython docs aren't in the sage reference manual? > As I said before, cython and cysignals don't do anything for math directly so they are a special case. In fact, they shouldn't be in the main Sage docs even if they were part of Sage. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
It would be nice if (1) Sage library consists of core + optional parts (2) An optional part is dependent on core + other optional parts (3) Optional parts to install can be selected at build time. (4) An optional part has its own documentation, but accessible through a single interface in Sage. For example, people may develop an optional part for function fields in a separate (GitHub) repository and discuss on a separate (Google) group. Perhaps this functionality can be provided by just "sage -pip install ..." as suggested above. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
Hi, On Thu, Apr 07, 2016 at 05:36:21PM -0700, William Stein wrote: > The sage reference manual is way too big already. Building it is a > high source of friction and complaints by users. It's worth (and also fun) to point out there that the second major topic at Sage Days 77 here after packaging and modularization et Sphinx and the compilation of the doc. We are still at cleaning the mess before going forward, but some experiment show that there is a good hope to have a better doc build system. Cheers, Florent -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Thursday, April 7, 2016, Volker Braun wrote: > On Friday, April 8, 2016 at 1:26:07 AM UTC+2, William wrote: >> >> This thread is first and foremost about reducing the friction involved in >> making code that depends on the Sage distribution available to the world. >> > > Whats wrong with the obvious solution: make it a Python package (basically > add setup.py) and then "sage -pip install > https://github.com/vbraun/awesomepackage.git";. Clearly we could have more > documentation for how to write the setup.py or some package directory > service. But really it is already a one-liner. > That is exactly what I'm proposing. However... > > Regarding your car analogy, sometimes it is far better if we can factor of >> some pieces of sage as separate libraries that we use, but which are also >> used outside of sage. For example, cysignals (I think from Malb). The >> broader community benefits and we also benefit since more people care about >> the code and work on it. It's a win-win. >> > > Yes but cython and cysignals don't do anything for math directly, they are > pure dependencies. By contrast, lets say we break out all elliptic curve > stuff into a separate package. Just like with cysignals, this will remove > all elliptic curve documentation from the Sage reference manual. Still a > win-win? > > 1. Not necessarily true. 2. Is it so bad that the Cython docs aren't in the sage reference manual? The sage reference manual is way too big already. Building it is a high source of friction and complaints by users. > > > > >> Remember that Cython used to be part of Sage and be called SageX... >> Making it separate was good for everybody. >> >> - William >> >> >> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sage-devel" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sage-devel+unsubscr...@googlegroups.com. >>> To post to this group, send email to sage-devel@googlegroups.com. >>> Visit this group at https://groups.google.com/group/sage-devel. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> Sent from my massive iPhone 6 plus. >> > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
Correction: "sage -pip install git+https://github.com/vbraun/awesomepackage.git"; -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 1:26:07 AM UTC+2, William wrote: > > This thread is first and foremost about reducing the friction involved in > making code that depends on the Sage distribution available to the world. > Whats wrong with the obvious solution: make it a Python package (basically add setup.py) and then "sage -pip install https://github.com/vbraun/awesomepackage.git";. Clearly we could have more documentation for how to write the setup.py or some package directory service. But really it is already a one-liner. Regarding your car analogy, sometimes it is far better if we can factor of > some pieces of sage as separate libraries that we use, but which are also > used outside of sage. For example, cysignals (I think from Malb). The > broader community benefits and we also benefit since more people care about > the code and work on it. It's a win-win. > Yes but cython and cysignals don't do anything for math directly, they are pure dependencies. By contrast, lets say we break out all elliptic curve stuff into a separate package. Just like with cysignals, this will remove all elliptic curve documentation from the Sage reference manual. Still a win-win? > Remember that Cython used to be part of Sage and be called SageX... > Making it separate was good for everybody. > > - William > > > >> -- >> You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-devel+unsubscr...@googlegroups.com. >> To post to this group, send email to sage-devel@googlegroups.com. >> Visit this group at https://groups.google.com/group/sage-devel. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Sent from my massive iPhone 6 plus. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 07, 2016 at 03:58:43PM -0700, William Stein wrote: > Cool! If there's any summary of discussions or questions, etc., let me know. We are taking notes in a couple of pads, including: https://www.lri.fr/etherpad/p/sage-days77-packaging This will end up after the Sage Days on the wiki or something similar (need to check whether there already is a page for this). Cheers, Nicolas -- Nicolas M. Thiéry "Isil" http://Nicolas.Thiery.name/ -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: how we develop sage
On Thursday, April 7, 2016, Volker Braun wrote: > On Friday, April 8, 2016 at 12:19:57 AM UTC+2, William wrote: >> >> > You mean like in the Linux kernel, which uses a single monolithic git >> > repository? >> I think you are being sarcastic. > > > I'm only partially kidding. Why is the kernel not a collection of > packages? Because nobody wants half a kernel, and because the kernel > developers don't want to make promises about internal APIs. Similarly, I > want a car but please leave out the engine and two of the four wheels said > nobody ever. > > >> There are very good reasons to supporting both modularization > > > And there are reasons against as well. Its a tool, and with every tool > there are pros and cons. The old adage about the right tool for the right > job still holds. > > The only thing that I'm convinced is that going around and claiming that > modularization is the solution of all our problems (without making a > concrete proposal about what and how) is not productive. I agree that our > current dependency installation system sucks (but are we even talking about > that or do you mean the Sage library). > > You seem to think that pypi/whl is a suitable venue for binary > distribution. As I mentioned before, the author of whl doesn't think so for > what would be our use case. Of course its trivial to publish a pure Python > package on pypi. So a sagemath pypi package that just assumes that you have > a dozen obscure math libraries (+headers) on you system to cythonize > against? Is that what this thread is even about? > This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world. Based on feedback I get from users, this friction is a very, very significant problem to the growth of Sage. There are projects with healthy package ecosystems. Examples include: tex, R, and Python. Often people write code for sage that sits in some branch on trac used by almost nobody for years and years. Just touching or working on that code involves way too much friction. In many cases that code would be much better placed in its own separate Python library. When it is fully up to snuff, we could make that Python library a standard part of the sage distribution of we want. Regarding your car analogy, sometimes it is far better if we can factor of some pieces of sage as separate libraries that we use, but which are also used outside of sage. For example, cysignals (I think from Malb). The broader community benefits and we also benefit since more people care about the code and work on it. It's a win-win. Remember that Cython used to be part of Sage and be called SageX... Making it separate was good for everybody. - William > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com > > . > To post to this group, send email to sage-devel@googlegroups.com > . > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- Sent from my massive iPhone 6 plus. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thursday, April 7, 2016 at 11:19:57 PM UTC+1, William wrote: > > On Wed, Apr 6, 2016 at 12:51 AM, Volker Braun > wrote: > > On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote: > >> > >> [...] toward standard open source practices. > > > > > > You mean like in the Linux kernel, which uses a single monolithic git > > repository? > > I think you are being sarcastic. It's hard to tell on the internet. > > > Really, modularization is not a useful goal in of itself. And it comes > with > > its own sets of issues, see the left-pad fiasco last week when the > nodejs > > clown boat caught fire. > > There are very good reasons to supporting both modularization and much > more standard approaches to packaging. It's unbelievable to me that > somebody as technically smart as you and knowledgeable about sage and > software doesn't see this as obvious. Since -- based on responses -- > almost nobody else in our community seems to get this either, I'll > just forge on and come up with the future approach to sage development > myself (even though I don't want to). > while better packaging would be great, and ability not to install (or uninstall) parts that one never uses (e.g., as far as I am concerned, I don't need finances/, games/), I fear that "full modularisation" efforts in case of "core" Sage are akin to surgery attempts to separate siamese twins. I cannot imagine separating groups, NT, algebra, linear algebra, symbolics, calculus, combinatorics/graph theory; they are too much interdependent for this. IMHO things like improving the build system are more urgent (and improving the build system would lessen the feel that Sage is too big...) > William > > > > > You can do "import sage" right now, it works. The question is, what do > you > > expect it to do. The basic dichotomy is > > > > * either this imports something like the "undergrad curriculum", all > kinds > > of rings and groups and plotting and ... (we are basically here, and > this > > necessarily has a longish list of dependencies). > > > > * or this imports some piece of Sage (integers?) that is super-fast to > > compile. But you want elliptic curves? Never heard of that, maybe if you > > install the sage-elliptic-curves (or maybe the competing and > incompatible > > sage-riemann-surface). > > > > As long as the goal of "import sage" is to give you something like the > > feature set of Sage right now we don't benefit from modularization. That > is > > just a tautology, the goal is just not a modular one. We'd just shoot > > ourselves in the foot if we split things into multiple interdependent > > packages that then must be upgraded in lockstep. > > > > It would still be useful to straighten out the Sage startup circular > import > > game, have some core feature set (~= sage.rings) that must be importable > > (and usable) by itself. This could be shipped as a separate package even > if > > its not a separate repository. It just requires tooling to keep > dependencies > > straight. And as I mentioned before, Python3 has better namespace > package > > support... > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Friday, April 8, 2016 at 12:19:57 AM UTC+2, William wrote: > > > You mean like in the Linux kernel, which uses a single monolithic git > > repository? > I think you are being sarcastic. I'm only partially kidding. Why is the kernel not a collection of packages? Because nobody wants half a kernel, and because the kernel developers don't want to make promises about internal APIs. Similarly, I want a car but please leave out the engine and two of the four wheels said nobody ever. > There are very good reasons to supporting both modularization And there are reasons against as well. Its a tool, and with every tool there are pros and cons. The old adage about the right tool for the right job still holds. The only thing that I'm convinced is that going around and claiming that modularization is the solution of all our problems (without making a concrete proposal about what and how) is not productive. I agree that our current dependency installation system sucks (but are we even talking about that or do you mean the Sage library). You seem to think that pypi/whl is a suitable venue for binary distribution. As I mentioned before, the author of whl doesn't think so for what would be our use case. Of course its trivial to publish a pure Python package on pypi. So a sagemath pypi package that just assumes that you have a dozen obscure math libraries (+headers) on you system to cythonize against? Is that what this thread is even about? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 7, 2016 at 3:41 PM, Nicolas M. Thiery wrote: > On Thu, Apr 07, 2016 at 03:19:14PM -0700, William Stein wrote: >> There are very good reasons to supporting both modularization and much >> more standard approaches to packaging. ... Since -- based on responses -- >> almost nobody else in our community seems to get this either, > > For the record, modularity is a major topic of the Sage Days we are > having right now (although the focus has been more about modularity at > the granularity of the Sage library w.r.t its dependencies than within > the Sage library): > > https://wiki.sagemath.org/days77 Cool! If there's any summary of discussions or questions, etc., let me know. I think a very useful sprint project for such a workshop would be to make a Sage-friendly version of this: https://python-packaging.readthedocs.org/en/latest/minimal.html Some issues are: - How to doctest code with "sage:" prompts that isn't in the sage library (basically, you have to be sure to do "sage -t --force-lib " and explicitly import from the Python library you're making in the docstrings. - Cython: relying on pxi/pxd files from the sage library. - Discussion about how to import functionality from the Sage library into a library. There could be a package (hosted on pip) called "sagemath" that other packages depend on. It could provide some helper functionality for the above. I make no claims to have thought anything through yet!I did however spend an hour playing around with this problem yesterday, which resulted in https://github.com/williamstein/sage_modabvar https://pypi.python.org/pypi/sage_modabvar/0.1 which is far from done. I just did a very tiny bit of initial work on removing this "big" pure-Python modular abelian varieties package from Sage and making it a standalone module. Thanks, William > > Cheers, > Nicolas > -- > Nicolas M. Thiéry "Isil" > http://Nicolas.Thiery.name/ > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- William (http://wstein.org) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: how we develop sage
On Thu, Apr 07, 2016 at 03:19:14PM -0700, William Stein wrote: > There are very good reasons to supporting both modularization and much > more standard approaches to packaging. ... Since -- based on responses -- > almost nobody else in our community seems to get this either, For the record, modularity is a major topic of the Sage Days we are having right now (although the focus has been more about modularity at the granularity of the Sage library w.r.t its dependencies than within the Sage library): https://wiki.sagemath.org/days77 Cheers, Nicolas -- Nicolas M. Thiéry "Isil" http://Nicolas.Thiery.name/ -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
