subject:"\[sage\-devel\] Re\: \[Proposal\] allow standard packages to be pip packages, reduce source tarball size"

Blocking on GitHub, I presume, is due to disagreements on a number of topics, 
including the topic discussed in this thread. So it's related to the topic, and 
personal only as it was done by a person, not by an AI.



On 27 February 2024 22:17:50 GMT, John H Palmieri  
wrote:
>That's called "whataboutism". Invoking what you consider inappropriate 
>behavior by others is not relevant. Please stay on topic, and please follow 
>Sage's code of conduct in your posts.
>
>On Tuesday, February 27, 2024 at 1:01:25 PM UTC-8 Dima Pasechnik wrote:
>
>>
>>
>> On 27 February 2024 20:44:50 GMT, John H Palmieri  
>> wrote:
>> >Sentences like "At the moment you are actively breaking down the precious 
>> >project fabric, all in the name of you having your way" are personal 
>> >attacks. Please stop.
>>
>> Blocking on GitHub members of the project is not a personal attack? 
>> Of course it is.
>>
>>
>> >
>> >On Tuesday, February 27, 2024 at 12:36:44 PM UTC-8 Dima Pasechnik wrote:
>> >
>> >>
>> >>
>> >> On 27 February 2024 19:37:31 GMT, Matthias Koeppe  
>>
>> >> wrote:
>> >> >On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri 
>> wrote:
>> >> >
>> >> >A pretty safe second choice would be to have "make download" also 
>> >> download 
>> >> >the relevant files for pip installation and tell pip where to find 
>> them. 
>> >> If 
>> >> >we implemented this second choice [...]
>> >> >
>> >> >
>> >> >The problem is that such tooling, even if "trivial", would need to be 
>> >> >implemented, tested, and maintained as well. And typically this just 
>> does 
>> >> >not work when there is no developer who is actually interested in 
>> using 
>> >> it.
>> >>
>> >> It is a largely artificially invented, by you, problem, to shoot my 
>> >> proposal down. Besides, these tools are trivial to build and maintain, 
>> much 
>> >> easier than your ever growing and breaking maze of packages we don't 
>> even 
>> >> need to vendor.
>> >>
>> >> At the moment you are actively breaking down the precious project 
>> fabric, 
>> >> all in the name of you having your way: 
>> >>
>> >> you blocked me (without bothering yo even tell me) and perhaps other 
>> >> developers on GitHub, meaning that you effectively want to shut me up.
>> >> (Well, I had no other choice but to block you too, as a countermeasure; 
>> I 
>> >> installed this block about 12:00 GMT, today.)
>> >>
>> >> Of course it's easy to skip this message. But tomorrow it might be you 
>> who 
>> >> Matthias might block, for disagreements with him.
>> >> Why is this tolerated? This is a naked attempt to shut down the 
>> opposition.
>> >>
>> >>
>> >> >
>> >> >We are better off with improving the tooling that we already know 
>> *will* 
>> >> >continue to be used: Namely the tooling for creating and maintaining 
>> our 
>> >> >metadata in build/pkgs.
>> >>
>> >> Or it will be discarded as useless, because doing things the way 
>> projects 
>> >> like scipy do is better.
>> >>
>> >> Your package tooling is makework, repeating with worse tools what 
>> already 
>> >> is done by Conda, Homebrew, Linux distros, etc. We are mainly a maths 
>> >> project, not a distro project. Let us stay this way, and actually do 
>> more 
>> >> maths and less distro-like stuff.
>> >>
>> >> Dima
>> >>
>> >>
>> >> > Issues such as:
>> >> >- https://github.com/sagemath/sage/issues/36356, 
>> >> >- https://github.com/sagemath/sage/pull/37322, 
>> >> >- https://github.com/sagemath/sage/issues/37323, 
>> >> >- https://github.com/sagemath/sage/issues/37314
>> >> >
>> >>
>> >
>>
>
>-- 
>You received this message because you are subscribed to the Google Groups 
>"sage-devel" group.
>To unsubscribe from this group and stop receiving emails from it, send an 
>email to sage-devel+unsubscr...@googlegroups.com.
>To view this discussion on the web visit 
>https://groups.google.com/d/msgid/sage-devel/3f85faf4-840a-4965-bd81-797f2733843fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6D2FFF01-A6CB-4398-87A7-171B07D521FB%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-27 Thread John H Palmieri

That's called "whataboutism". Invoking what you consider inappropriate 
behavior by others is not relevant. Please stay on topic, and please follow 
Sage's code of conduct in your posts.

On Tuesday, February 27, 2024 at 1:01:25 PM UTC-8 Dima Pasechnik wrote:

>
>
> On 27 February 2024 20:44:50 GMT, John H Palmieri  
> wrote:
> >Sentences like "At the moment you are actively breaking down the precious 
> >project fabric, all in the name of you having your way" are personal 
> >attacks. Please stop.
>
> Blocking on GitHub members of the project is not a personal attack? 
> Of course it is.
>
>
> >
> >On Tuesday, February 27, 2024 at 12:36:44 PM UTC-8 Dima Pasechnik wrote:
> >
> >>
> >>
> >> On 27 February 2024 19:37:31 GMT, Matthias Koeppe  
>
> >> wrote:
> >> >On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri 
> wrote:
> >> >
> >> >A pretty safe second choice would be to have "make download" also 
> >> download 
> >> >the relevant files for pip installation and tell pip where to find 
> them. 
> >> If 
> >> >we implemented this second choice [...]
> >> >
> >> >
> >> >The problem is that such tooling, even if "trivial", would need to be 
> >> >implemented, tested, and maintained as well. And typically this just 
> does 
> >> >not work when there is no developer who is actually interested in 
> using 
> >> it.
> >>
> >> It is a largely artificially invented, by you, problem, to shoot my 
> >> proposal down. Besides, these tools are trivial to build and maintain, 
> much 
> >> easier than your ever growing and breaking maze of packages we don't 
> even 
> >> need to vendor.
> >>
> >> At the moment you are actively breaking down the precious project 
> fabric, 
> >> all in the name of you having your way: 
> >>
> >> you blocked me (without bothering yo even tell me) and perhaps other 
> >> developers on GitHub, meaning that you effectively want to shut me up.
> >> (Well, I had no other choice but to block you too, as a countermeasure; 
> I 
> >> installed this block about 12:00 GMT, today.)
> >>
> >> Of course it's easy to skip this message. But tomorrow it might be you 
> who 
> >> Matthias might block, for disagreements with him.
> >> Why is this tolerated? This is a naked attempt to shut down the 
> opposition.
> >>
> >>
> >> >
> >> >We are better off with improving the tooling that we already know 
> *will* 
> >> >continue to be used: Namely the tooling for creating and maintaining 
> our 
> >> >metadata in build/pkgs.
> >>
> >> Or it will be discarded as useless, because doing things the way 
> projects 
> >> like scipy do is better.
> >>
> >> Your package tooling is makework, repeating with worse tools what 
> already 
> >> is done by Conda, Homebrew, Linux distros, etc. We are mainly a maths 
> >> project, not a distro project. Let us stay this way, and actually do 
> more 
> >> maths and less distro-like stuff.
> >>
> >> Dima
> >>
> >>
> >> > Issues such as:
> >> >- https://github.com/sagemath/sage/issues/36356, 
> >> >- https://github.com/sagemath/sage/pull/37322, 
> >> >- https://github.com/sagemath/sage/issues/37323, 
> >> >- https://github.com/sagemath/sage/issues/37314
> >> >
> >>
> >
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3f85faf4-840a-4965-bd81-797f2733843fn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 27 February 2024 20:44:50 GMT, John H Palmieri  
wrote:
>Sentences like "At the moment you are actively breaking down the precious 
>project fabric, all in the name of you having your way" are personal 
>attacks. Please stop.

Blocking on GitHub members of the project is not a personal attack? 
Of course it is.


>
>On Tuesday, February 27, 2024 at 12:36:44 PM UTC-8 Dima Pasechnik wrote:
>
>>
>>
>> On 27 February 2024 19:37:31 GMT, Matthias Koeppe  
>> wrote:
>> >On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri wrote:
>> >
>> >A pretty safe second choice would be to have "make download" also 
>> download 
>> >the relevant files for pip installation and tell pip where to find them. 
>> If 
>> >we implemented this second choice [...]
>> >
>> >
>> >The problem is that such tooling, even if "trivial", would need to be 
>> >implemented, tested, and maintained as well. And typically this just does 
>> >not work when there is no developer who is actually interested in using 
>> it.
>>
>> It is a largely artificially invented, by you, problem, to shoot my 
>> proposal down. Besides, these tools are trivial to build and maintain, much 
>> easier than your ever growing and breaking maze of packages we don't even 
>> need to vendor.
>>
>> At the moment you are actively breaking down the precious project fabric, 
>> all in the name of you having your way: 
>>
>> you blocked me (without bothering yo even tell me) and perhaps other 
>> developers on GitHub, meaning that you effectively want to shut me up.
>> (Well, I had no other choice but to block you too, as a countermeasure; I 
>> installed this block about 12:00 GMT, today.)
>>
>> Of course it's easy to skip this message. But tomorrow it might be you who 
>> Matthias might block, for disagreements with him.
>> Why is this tolerated? This is a naked attempt to shut down the opposition.
>>
>>
>> >
>> >We are better off with improving the tooling that we already know *will* 
>> >continue to be used: Namely the tooling for creating and maintaining our 
>> >metadata in build/pkgs.
>>
>> Or it will be discarded as useless, because doing things the way projects 
>> like scipy do is better.
>>
>> Your package tooling is makework, repeating with worse tools what already 
>> is done by Conda, Homebrew, Linux distros, etc. We are mainly a maths 
>> project, not a distro project. Let us stay this way, and actually do more 
>> maths and less distro-like stuff.
>>
>> Dima
>>
>>
>> > Issues such as:
>> >- https://github.com/sagemath/sage/issues/36356, 
>> >- https://github.com/sagemath/sage/pull/37322, 
>> >- https://github.com/sagemath/sage/issues/37323, 
>> >- https://github.com/sagemath/sage/issues/37314
>> >
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/4465036B-3AF2-45ED-BCAA-9903B13757ED%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 27 February 2024 20:21:26 GMT, Nils Bruin  wrote:
>On Tuesday 27 February 2024 at 10:50:55 UTC-8 John H Palmieri wrote:
>
>
>As Nathan points out, this will likely lead to instability. Someone will 
>upgrade some component, and most of the time that will be fine, but 
>occasionally it will break something on some platform, and it could be 
>annoying to track down the cause. If this leads to Sage failing to build, 
>that's not great, but it would be *far worse* if Sage built and ran but 
>produced some mathematically incorrect answers. Being able to control all 
>of the versions means that our doctests are pretty robust. If we really 
>want to go down the road of unpinning version requirements, I propose that 
>we *always* pin version requirements for the mathematical components of 
>Sage. If Jupyter or Sphinx doesn't work right, it doesn't affect the 
>mathematics, but if linbox or pari don't work right (or ore_algebra, if you 
>want a pip package), people could be getting different answers on different 
>platforms and we might not know about it for a while. To maintain the 
>mathematical integrity of the project, we should keep very careful control 
>of the mathematical components of Sage.
>
>+1 to this. There is another difference: The jupyter notebook server - 
>notebook kernel interface is not a binary one: it's a protocol. As such, 
>it's hopefully narrower in its definition and a bit more stable. So 
>offering that sagemath offers a notebook kernel that a separately 
>distributed notebook server can connect to is a different dependency 
>structure than depending on libraries being provided. There are some super 
>stable libraries for which external dependencies are OK, but generally 
>(especially components under active development) they are a lot more 
>sensitive.
>
>For full functionality we depend on more than vanilla jupyter notebook, 
>though: at some point, jupyter notebook server shipped with a pared-down 
>mathjax and I have not been able to to get proper mathjax working in my 
>system-provided jupyter notebook. 

The reason for having a bad integration with upstream Jupyter is simple - we 
haven't really tried to do it properly, that's all. 

And I am not able to get Sage's Jupyter properly working. Also people here 
complain that they can't use Sage's Jupyter offline (it's cause we keep on 
using Mathjax2, for the sole reason that long formulas dont wrap around, 
instead there is a slider to view them). So if one wants offline MathJax, e.g. 
for a presentation, Sage's Jupyter is useless.

With an ability to have standard pip packages, this would be something to look 
at - cause without this ability it's always some ad hoc non-official stuff, may 
work, or not, no one  cares much.

> We have other components, such as 
>documentation and various plug-ins that we need in jupyter notebook too. 
>This has made the shipped-with-sage jupyter notebook server more reliable 
>to me, even if I'd prefer the system notebook server if that would be 
>reliable to get working.
>
>Sphinx is another part of such tooling: it should be able to just read and 
>process markup. We tend to not ship a latex installation with our 
>preprints.
>  However, I don't have experience with how stable sphinx on its 
>own is to judge how much trouble one would get from relying on an external 
>sphinx.

If you pin a version, or even if you don't, there should be no difference 
between what we install, and what comes from pip install. Gentoo provides a 
very modern upstream Sphinx, and it works with Sage, for quite some time.


>  Personally, I don't bother building the sage documentation and rely 
>on what is available online instead, so that one wouldn't affect me.  
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/978D1F98-CC2C-404A-84DB-110025C7FDB3%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-27 Thread John H Palmieri

Sentences like "At the moment you are actively breaking down the precious 
project fabric, all in the name of you having your way" are personal 
attacks. Please stop.

On Tuesday, February 27, 2024 at 12:36:44 PM UTC-8 Dima Pasechnik wrote:

>
>
> On 27 February 2024 19:37:31 GMT, Matthias Koeppe  
> wrote:
> >On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri wrote:
> >
> >A pretty safe second choice would be to have "make download" also 
> download 
> >the relevant files for pip installation and tell pip where to find them. 
> If 
> >we implemented this second choice [...]
> >
> >
> >The problem is that such tooling, even if "trivial", would need to be 
> >implemented, tested, and maintained as well. And typically this just does 
> >not work when there is no developer who is actually interested in using 
> it.
>
> It is a largely artificially invented, by you, problem, to shoot my 
> proposal down. Besides, these tools are trivial to build and maintain, much 
> easier than your ever growing and breaking maze of packages we don't even 
> need to vendor.
>
> At the moment you are actively breaking down the precious project fabric, 
> all in the name of you having your way: 
>
> you blocked me (without bothering yo even tell me) and perhaps other 
> developers on GitHub, meaning that you effectively want to shut me up.
> (Well, I had no other choice but to block you too, as a countermeasure; I 
> installed this block about 12:00 GMT, today.)
>
> Of course it's easy to skip this message. But tomorrow it might be you who 
> Matthias might block, for disagreements with him.
> Why is this tolerated? This is a naked attempt to shut down the opposition.
>
>
> >
> >We are better off with improving the tooling that we already know *will* 
> >continue to be used: Namely the tooling for creating and maintaining our 
> >metadata in build/pkgs.
>
> Or it will be discarded as useless, because doing things the way projects 
> like scipy do is better.
>
> Your package tooling is makework, repeating with worse tools what already 
> is done by Conda, Homebrew, Linux distros, etc. We are mainly a maths 
> project, not a distro project. Let us stay this way, and actually do more 
> maths and less distro-like stuff.
>
> Dima
>
>
> > Issues such as:
> >- https://github.com/sagemath/sage/issues/36356, 
> >- https://github.com/sagemath/sage/pull/37322, 
> >- https://github.com/sagemath/sage/issues/37323, 
> >- https://github.com/sagemath/sage/issues/37314
> >
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/15220b4c-09f8-4408-9152-816ef45b4261n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On 27 February 2024 19:37:31 GMT, Matthias Koeppe  
wrote:
>On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri wrote:
>
>A pretty safe second choice would be to have "make download" also download 
>the relevant files for pip installation and tell pip where to find them. If 
>we implemented this second choice [...]
>
>
>The problem is that such tooling, even if "trivial", would need to be 
>implemented, tested, and maintained as well. And typically this just does 
>not work when there is no developer who is actually interested in using it.

It is a largely artificially invented, by you, problem, to shoot my proposal 
down. Besides, these tools are trivial to build and maintain, much easier than 
your ever growing and breaking maze of packages we don't even need to vendor.

At the moment you are actively breaking down the precious project fabric, all 
in the name of you having your way: 

you blocked me (without bothering yo even tell me) and perhaps other developers 
on GitHub, meaning that you effectively want to shut me up.
(Well, I had no other choice but to block you too, as a countermeasure; I 
installed this block  about 12:00 GMT, today.)

Of course it's easy to skip this message. But tomorrow it might be you who 
Matthias might block, for disagreements with him.
Why is this tolerated? This is a naked attempt to shut down the opposition.

>
>We are better off with improving the tooling that we already know *will* 
>continue to be used: Namely the tooling for creating and maintaining our 
>metadata in build/pkgs.

Or it will be discarded as useless, because doing things the way projects like 
scipy do is better.

Your package tooling is makework, repeating with worse tools what already is 
done by Conda, Homebrew, Linux distros, etc. We are mainly a maths project, not 
a distro project. Let us stay this way, and actually do more maths and less 
distro-like stuff.

Dima

>  Issues such as:
>- https://github.com/sagemath/sage/issues/36356, 
>- https://github.com/sagemath/sage/pull/37322, 
>- https://github.com/sagemath/sage/issues/37323, 
>- https://github.com/sagemath/sage/issues/37314
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/06BFE7EF-F2E0-4072-B8A0-390DA9947AE6%40gmail.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-27 Thread Nils Bruin

On Tuesday 27 February 2024 at 10:50:55 UTC-8 John H Palmieri wrote:

As Nathan points out, this will likely lead to instability. Someone will
upgrade some component, and most of the time that will be fine, but
occasionally it will break something on some platform, and it could be
annoying to track down the cause. If this leads to Sage failing to build,
that's not great, but it would be *far worse* if Sage built and ran but
produced some mathematically incorrect answers. Being able to control all
of the versions means that our doctests are pretty robust. If we really
want to go down the road of unpinning version requirements, I propose that
we *always* pin version requirements for the mathematical components of
Sage. If Jupyter or Sphinx doesn't work right, it doesn't affect the
mathematics, but if linbox or pari don't work right (or ore_algebra, if you
want a pip package), people could be getting different answers on different
platforms and we might not know about it for a while. To maintain the
mathematical integrity of the project, we should keep very careful control
of the mathematical components of Sage.

+1 to this. There is another difference: The jupyter notebook server -
notebook kernel interface is not a binary one: it's a protocol. As such,
it's hopefully narrower in its definition and a bit more stable. So
offering that sagemath offers a notebook kernel that a separately
distributed notebook server can connect to is a different dependency
structure than depending on libraries being provided. There are some super
stable libraries for which external dependencies are OK, but generally
(especially components under active development) they are a lot more
sensitive.

For full functionality we depend on more than vanilla jupyter notebook,
though: at some point, jupyter notebook server shipped with a pared-down
mathjax and I have not been able to to get proper mathjax working in my
system-provided jupyter notebook. We have other components, such as
documentation and various plug-ins that we need in jupyter notebook too.
This has made the shipped-with-sage jupyter notebook server more reliable
to me, even if I'd prefer the system notebook server if that would be
reliable to get working.

Sphinx is another part of such tooling: it should be able to just read and
process markup. We tend to not ship a latex installation with our
preprints. However, I don't have experience with how stable sphinx on its
own is to judge how much trouble one would get from relying on an external
sphinx. Personally, I don't bother building the sage documentation and rely
on what is available online instead, so that one wouldn't affect me.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/88accdbb-7d2a-4860-8791-787bf52a8a89n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 27 February 2024 18:50:55 GMT, John H Palmieri  
wrote:
>Regarding the proposal to allow standard packages to be pip packages, no 
>one really knows how much people rely on the all-in-one tarball that we 
>currently distribute. No one really knows how often the "make download" 
>option is used for people who just clone the git repo and want to do all of 
>their downloads at once. Since we don't know, the absolute safest course of 
>action is to not change anything, but maybe that's too conservative. A 
>pretty safe second choice would be to have "make download" also download 
>the relevant files for pip installation and tell pip where to find them. If 
>we implemented this second choice, I would support this aspect of Dima's 
>proposal.

At the moment we are talking about using this option,
have pytest* and build, as standard pip packages, whereas they are now optional 
pip packages.

This is all by now. We lived for years with them as optional packages, nobody 
had problems with them, why would they all of a sudden started causing problems?

>
>My impression from Dima's posts is that he would like us to frequently not 
>provide version information for pip packages. Here are some of my thoughts:
>
>As Nathan points out, this will likely lead to instability. 


No known to me project vendors pytest on the basis that doing otherwise would 
"lead to instability".

Let us be practical - there is no reason whatsoever to vendor pytest*.
Therefore standard pip packages should be allowed.

No known to me project vendors jupyterlab, either.

This is just FUD that a well maintained package like 
pytest must be vendored.
It's as believable as saying that tar must be vendored, cause a version not 
tested might lead to maths errors.


> Someone will 
>upgrade some component, and most of the time that will be fine, but 
>occasionally it will break something on some platform, and it could be 
>annoying to track down the cause. If this leads to Sage failing to build, 
>that's not great, but it would be *far worse* if Sage built and ran but 
>produced some mathematically incorrect answers.
> Being able to control all 
>of the versions means that our doctests are pretty robust. If we really 
>want to go down the road of unpinning version requirements, I propose that 
>we *always* pin version requirements for the mathematical components of 
>Sage.

We can still pin version requirements on pip packages. And indeed for maths 
packages,
a handful of these which are maths, we can always do this, why not?

> If Jupyter or Sphinx doesn't work right, it doesn't affect the 
>mathematics, but if linbox or pari don't work right (or ore_algebra, if you 
>want a pip package), people could be getting different answers on different 
>platforms and we might not know about it for a while. To maintain the 
>mathematical integrity of the project, we should keep very careful control 
>of the mathematical components of Sage.

Sure, I am fine with this. All what my proposal meant to do is to be able to 
get rid of lots of vendored packages which have nothing to do with maths, they 
are just guts of pytest, Jupyter, other common place packages.


>
>
>On Saturday, February 24, 2024 at 3:40:02 PM UTC-8 Matthias Koeppe wrote:
>
>> On Monday, February 12, 2024 at 6:42:07 PM UTC-8 Matthias Koeppe wrote:
>>
>> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>>
>>
>> *Proposed action items: *
>> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
>> that "git clone" is described as the primary way to obtain the Sage 
>> sources. That the big release tarball is available can be a footnote in the 
>> Installation Guide (
>> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>>  
>> for the limited no-internet connectivity use case.
>>
>>
>> That's now https://github.com/sagemath/sage/pull/37309 (needs review)
>>
>>
>> Thanks for all comments on the PR. Ready for final review. 
>>  
>>
>>  *B. *Likewise, get rid of all of these "Download Sage source code" pages 
>> (https://www.sagemath.org/download-source.html, 
>> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
>> from the Sage website. 
>>
>> That's now https://github.com/sagemath/website/pull/466
>>
>>
>> This one has already been merged, thanks, Harald! 
>> The "Download" menu no longer sends people to the tarball download pages. 
>> The pages are, of course, still there, and links from the installation 
>> guide point to them.
>>
>> [image: Screenshot 2024-02-24 at 3.34.47 PM.png]
>>
>> On Tuesday, February 20, 2024 at 6:44:32 PM UTC-8 Matthias Koeppe wrote:
>>
>> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:
>>
>> Prompted by the discussion of space use on the *local machines* of users 
>> and developers, I propose another item in addition to A and B:
>>
>> *C. Advertise use of "git worktree" and recommend symlinking the 
>> "upstream" directory.* For

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-27 Thread Matthias Koeppe

On Tuesday, February 27, 2024 at 10:50:55 AM UTC-8 John H Palmieri wrote:

A pretty safe second choice would be to have "make download" also download 
the relevant files for pip installation and tell pip where to find them. If 
we implemented this second choice [...]


The problem is that such tooling, even if "trivial", would need to be 
implemented, tested, and maintained as well. And typically this just does 
not work when there is no developer who is actually interested in using it.

We are better off with improving the tooling that we already know *will* 
continue to be used: Namely the tooling for creating and maintaining our 
metadata in build/pkgs. Issues such as:
- https://github.com/sagemath/sage/issues/36356, 
- https://github.com/sagemath/sage/pull/37322, 
- https://github.com/sagemath/sage/issues/37323, 
- https://github.com/sagemath/sage/issues/37314

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/0dc81e8a-d3c1-488f-9c53-580f21af2821n%40googlegroups.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-27 Thread John H Palmieri

Regarding the proposal to allow standard packages to be pip packages, no 
one really knows how much people rely on the all-in-one tarball that we 
currently distribute. No one really knows how often the "make download" 
option is used for people who just clone the git repo and want to do all of 
their downloads at once. Since we don't know, the absolute safest course of 
action is to not change anything, but maybe that's too conservative. A 
pretty safe second choice would be to have "make download" also download 
the relevant files for pip installation and tell pip where to find them. If 
we implemented this second choice, I would support this aspect of Dima's 
proposal.

My impression from Dima's posts is that he would like us to frequently not 
provide version information for pip packages. Here are some of my thoughts:

As Nathan points out, this will likely lead to instability. Someone will 
upgrade some component, and most of the time that will be fine, but 
occasionally it will break something on some platform, and it could be 
annoying to track down the cause. If this leads to Sage failing to build, 
that's not great, but it would be *far worse* if Sage built and ran but 
produced some mathematically incorrect answers. Being able to control all 
of the versions means that our doctests are pretty robust. If we really 
want to go down the road of unpinning version requirements, I propose that 
we *always* pin version requirements for the mathematical components of 
Sage. If Jupyter or Sphinx doesn't work right, it doesn't affect the 
mathematics, but if linbox or pari don't work right (or ore_algebra, if you 
want a pip package), people could be getting different answers on different 
platforms and we might not know about it for a while. To maintain the 
mathematical integrity of the project, we should keep very careful control 
of the mathematical components of Sage.

On Saturday, February 24, 2024 at 3:40:02 PM UTC-8 Matthias Koeppe wrote:

> On Monday, February 12, 2024 at 6:42:07 PM UTC-8 Matthias Koeppe wrote:
>
> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>
>
> *Proposed action items: *
> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
> that "git clone" is described as the primary way to obtain the Sage 
> sources. That the big release tarball is available can be a footnote in the 
> Installation Guide (
> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>  
> for the limited no-internet connectivity use case.
>
>
> That's now https://github.com/sagemath/sage/pull/37309 (needs review)
>
>
> Thanks for all comments on the PR. Ready for final review. 
>  
>
>  *B. *Likewise, get rid of all of these "Download Sage source code" pages 
> (https://www.sagemath.org/download-source.html, 
> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
> from the Sage website. 
>
> That's now https://github.com/sagemath/website/pull/466
>
>
> This one has already been merged, thanks, Harald! 
> The "Download" menu no longer sends people to the tarball download pages. 
> The pages are, of course, still there, and links from the installation 
> guide point to them.
>
> [image: Screenshot 2024-02-24 at 3.34.47 PM.png]
>
> On Tuesday, February 20, 2024 at 6:44:32 PM UTC-8 Matthias Koeppe wrote:
>
> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:
>
> Prompted by the discussion of space use on the *local machines* of users 
> and developers, I propose another item in addition to A and B:
>
> *C. Advertise use of "git worktree" and recommend symlinking the 
> "upstream" directory.* For testing a new release when you have an 
> existing clone of the repository, using "git clone" another time is 
> overkill as it creates another copy of the .git directory. And there is no 
> point in having multiple copies of the "upstream" directory, as the 
> filenames of the tarballs change whenever the contents change.
>
>
> That's now https://github.com/sagemath/sage/pull/37411 (needs review)
>
>
> This one has been positively reviewed. Thanks, Lorenz! 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/b448c1da-eb22-4402-aea7-09bec2ab88e9n%40googlegroups.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-22 Thread Gareth Ma

Can someone from this list confirm that the PR Matthias linked (#37411) is 
okay i.e. not the topic of the debate in this thread? The added 
documentation seems harmless to me, advertises using git worktree, and can 
be updated when the discussion in this thread has a conclusion e.g. if 
upstream is moved or something.

Also, I suggest everyone to take a short break from Sage infra issues and 
go for a hike when possible.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/ca9e531e-724b-4f38-8e36-7470ff0a5f0en%40googlegroups.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:

Prompted by the discussion of space use on the *local machines* of users 
and developers, I propose another item in addition to A and B:

*C. Advertise use of "git worktree" and recommend symlinking the "upstream" 
directory.* For testing a new release when you have an existing clone of 
the repository, using "git clone" another time is overkill as it creates 
another copy of the .git directory. And there is no point in having 
multiple copies of the "upstream" directory, as the filenames of the 
tarballs change whenever the contents change.


That's now https://github.com/sagemath/sage/pull/37411 (needs review)

(I've written it without introducing a new Sage command for now.)

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/44665827-048e-4cc7-a154-1d6459196633n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 20 February 2024 22:44:02 GMT, Matthias Koeppe  
wrote:
>On Tuesday, February 20, 2024 at 12:45:25 PM UTC-8 Dima Pasechnik wrote:
>
>On Tue, Feb 20, 2024 at 8:13 PM Matthias Koeppe  
>wrote:
>
>I have been doing the vast majority of this maintenance work in the past 4 
>years, and I have been improving the tooling to reduce the workload 
>associated with it -- and to make it more accessible to other contributors.
>
>
>testing and reviewing this endless stream of updates and new packages, etc. 
>is also work, a lot.
>
>
>I'll note that testing of most package upgrades has been automated by the 
>CI, specifically by my CI Linux Incremental workflow. Manual testing of 
>most of these upgrades is not necessary.
> 


>
>And you also are not shy in outsourcing your "vast majority" either.
>E.g. I got totally fed up with this at this point:
>https://github.com/sagemath/sage/pull/36777
>- where you kindly invited me to add few hundred text files by hand, 
>copy-pasting from the repology repo - all thanks to your truly wonderful 
>labour-reducing tooling... Wondeful tooling - homo sapience with vi and 
>mouse - very flexible, reliable.
>
>
>I can confirm that you seemed upset. But everything else is false.
>
>The project of supporting use of system site-packages via "./configure 
>--enable-system-site-packages" 
>(https://github.com/sagemath/sage/wiki/Sage-10.2-Release-Tour#configure---enable-system-site-packages-experimental)
> 
>is not mine. I have expressed my skepticism about it early on, but have 
>nevertheless supported its merge 
>(https://github.com/sagemath/sage/pull/36141) for experiments by the 
>interested parties, and in the interest of synergy/collaboration on 
>maintaining the "install-requires.txt" files, which have another use for 
>the metadata of our packages on PyPI. 
>
>This feature remains in experimental status, as it fails on numerous 
>platforms (see https://github.com/sagemath/sage/actions/runs/7894201915, 
>section "standard-sitepackages").
>
>When I saw what you were trying to do in your PR 
>https://github.com/sagemath/sage/pull/36777, trying to support use of 
>system Jupyter by means of this system-site-packages mechanism (and using 
>the very "micro-management of dependencies" that you are lamenting about 
>here), I suggested a better approach 
>in https://github.com/sagemath/sage/pull/36777#issuecomment-1831172595 and 
>https://github.com/sagemath/sage/pull/36777#issuecomment-1832426588. 
>Unfortunately you dismissed it without giving it much consideration.
>
>I then invited you to work on an Issue that would improve the tooling, 
>https://github.com/sagemath/sage/issues/36356, exactly to eliminate the 
>copy-pasting from the repology repo that you were doing, but you declined.
>

I basically don't want to collaborate with you, Matthias, because of your 
conduct. Sometimes I have to, but too often interactions with you piss me off 
too much, and then I just don't feel like keeping on. 


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/5FD4CF1A-A934-4814-9F11-9E927A6E0240%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Tuesday, February 20, 2024 at 12:45:25 PM UTC-8 Dima Pasechnik wrote:

On Tue, Feb 20, 2024 at 8:13 PM Matthias Koeppe
wrote:

I have been doing the vast majority of this maintenance work in the past 4
years, and I have been improving the tooling to reduce the workload
associated with it -- and to make it more accessible to other contributors.

testing and reviewing this endless stream of updates and new packages, etc.
is also work, a lot.

I'll note that testing of most package upgrades has been automated by the
CI, specifically by my CI Linux Incremental workflow. Manual testing of
most of these upgrades is not necessary.

And you also are not shy in outsourcing your "vast majority" either.
E.g. I got totally fed up with this at this point:
https://github.com/sagemath/sage/pull/36777
- where you kindly invited me to add few hundred text files by hand,
copy-pasting from the repology repo - all thanks to your truly wonderful
labour-reducing tooling... Wondeful tooling - homo sapience with vi and
mouse - very flexible, reliable.

I can confirm that you seemed upset. But everything else is false.

The project of supporting use of system site-packages via "./configure
--enable-system-site-packages"
(https://github.com/sagemath/sage/wiki/Sage-10.2-Release-Tour#configure---enable-system-site-packages-experimental)

is not mine. I have expressed my skepticism about it early on, but have
nevertheless supported its merge
(https://github.com/sagemath/sage/pull/36141) for experiments by the
interested parties, and in the interest of synergy/collaboration on
maintaining the "install-requires.txt" files, which have another use for
the metadata of our packages on PyPI.

This feature remains in experimental status, as it fails on numerous
platforms (see https://github.com/sagemath/sage/actions/runs/7894201915,
section "standard-sitepackages").

When I saw what you were trying to do in your PR
https://github.com/sagemath/sage/pull/36777, trying to support use of
system Jupyter by means of this system-site-packages mechanism (and using
the very "micro-management of dependencies" that you are lamenting about
here), I suggested a better approach
in https://github.com/sagemath/sage/pull/36777#issuecomment-1831172595 and
https://github.com/sagemath/sage/pull/36777#issuecomment-1832426588.
Unfortunately you dismissed it without giving it much consideration.

I then invited you to work on an Issue that would improve the tooling,
https://github.com/sagemath/sage/issues/36356, exactly to eliminate the
copy-pasting from the repology repo that you were doing, but you declined.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/094eaf14-2802-4f45-90f2-af9158e60ea3n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Tue, Feb 20, 2024 at 8:13 PM Matthias Koeppe 
wrote:

> On Tuesday, February 20, 2024 at 11:43:07 AM UTC-8 Dima Pasechnik wrote:
>
> On 20 February 2024 17:28:31 GMT, Matthias Koeppe 
> wrote:
> >On Tuesday, February 20, 2024 at 1:43:27 AM UTC-8 Dima Pasechnik wrote:
> >The number of dependencies has grown to the point it has gotten too hard
> to
> >maintain,
> >
> >No. It's easier than it has ever been in the past because of our improved
> >tooling.
>
> You keep adding more dependencies to justify ever growing tooling, it
> seems.
>
>
> No, Dima, that's absurd.
>
> I have been doing the vast majority of this maintenance work in the past 4
> years, and I have been improving the tooling to reduce the workload
> associated with it -- and to make it more accessible to other contributors.
>

testing and reviewing this endless stream of updates and new packages, etc.
is also work, a lot.
And you also are not shy in outsourcing your "vast majority" either.
E.g. I got totally fed up with this at this point:
https://github.com/sagemath/sage/pull/36777
- where you kindly invited me to add few hundred text files by hand,
copy-pasting from the repology repo - all thanks to your truly wonderful
labour-reducing tooling... Wondeful tooling - homo sapience with vi and
mouse - very flexible, reliable.
I repeat:
we don't need the guts of Jupyterlab etc in Sage repo, full stop - in fact
in this case just mentioned #36777 would be not needed.



> Source:
> echo "  commit | date | author | subject"; echo "  -- | -- | -- | --"; git
> --no-pager log --since 2020 --no-merges --format="  %h | %as | %<(24)%aN |
> %s" build/pkgs/*/{package-version.txt,spkg-*.in,patches}
>
> Only you know what it does and how to use it.
>
>
> It's well-documented in our Developer Guide (
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging
> ).
>
> I strongly encourage others to read it -- and welcome requests for
> improvements and PRs --  but I certainly can't force anyone to read it.
>
> What if we just don't want it?
>
>
> Who is "we"?
>

You'll know when we announce a fork. :-)


>
> I certainly don't care about the guts of Jupyterlab and Sphinx, I just
> want to use them.
>
>
> That's OK. That's what most users and developers do.
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/3cf3e70e-91c6-43f0-bc2a-bb633acc9b55n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2i0HjgLfGt4qbuC7N-ayUkMJO0KSLOU-mbG0xWr0%3D2Eg%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Tuesday, February 20, 2024 at 11:43:07 AM UTC-8 Dima Pasechnik wrote:

On 20 February 2024 17:28:31 GMT, Matthias Koeppe  
wrote: 
>On Tuesday, February 20, 2024 at 1:43:27 AM UTC-8 Dima Pasechnik wrote: 
>The number of dependencies has grown to the point it has gotten too hard 
to 
>maintain, 
> 
>No. It's easier than it has ever been in the past because of our improved 
>tooling. 

You keep adding more dependencies to justify ever growing tooling, it seems.

No, Dima, that's absurd.

I have been doing the vast majority of this maintenance work in the past 4 
years, and I have been improving the tooling to reduce the workload 
associated with it -- and to make it more accessible to other contributors.

Source:
echo "  commit | date | author | subject"; echo "  -- | -- | -- | --"; git 
--no-pager log --since 2020 --no-merges --format="  %h | %as | %<(24)%aN | 
%s" build/pkgs/*/{package-version.txt,spkg-*.in,patches}

Only you know what it does and how to use it.

It's well-documented in our Developer Guide 
(https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging). 

I strongly encourage others to read it -- and welcome requests for 
improvements and PRs --  but I certainly can't force anyone to read it.

What if we just don't want it?

Who is "we"?

I certainly don't care about the guts of Jupyterlab and Sphinx, I just want 
to use them.

That's OK. That's what most users and developers do.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3cf3e70e-91c6-43f0-bc2a-bb633acc9b55n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On 20 February 2024 17:28:31 GMT, Matthias Koeppe  
wrote:
>On Tuesday, February 20, 2024 at 1:43:27 AM UTC-8 Dima Pasechnik wrote:
>
>The number of dependencies has grown to the point it has gotten too hard to 
>maintain,
>
>
>No. It's easier than it has ever been in the past because of our improved 
>tooling.

You keep adding more dependencies to justify ever growing tooling, it seems. 
Only you know what it does and how to use it. 

What if we just don't want it? I certainly don't care about the guts of 
Jupyterlab and Sphinx, I just want to use them. You for some reason want to 
maintain a vendored version of them. 
Has it ever crossed your mind to ask whether we want to toil on this vendored 
version? IMO it is a waste of time and effort.

We might just do a fork of Sage, if you keep preventing reduction in number of 
dependencies, you know. Good luck then finding people willing to toil with 
these tools of yours, or without.

> 
>
>especially if one aims to support as many Python versions as we do. 
>These dependencies force one to have fragile, and temporary, 
>version-dependent workarounds in the configuration.
>
>
>What do you mean by that?

You perfectly know what I mean. There are ongoing debates on Sage's GitHub repo 
coming from various Python packages being obsoleted as projects and distros 
drop Python 3.9 and 3.10.
And you are actively preventing people to do work there, e.g. on #36753. 

>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/E66D7ABF-9287-4F7A-A34F-24F7B1398D7B%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Tuesday, February 20, 2024 at 1:43:27 AM UTC-8 Dima Pasechnik wrote:

The number of dependencies has grown to the point it has gotten too hard to 
maintain,


No. It's easier than it has ever been in the past because of our improved 
tooling.
 

especially if one aims to support as many Python versions as we do. 
These dependencies force one to have fragile, and temporary, 
version-dependent workarounds in the configuration.


What do you mean by that?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/986e90bc-19df-405b-bf7c-bd09f24a1949n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 20 February 2024 01:57:40 GMT, Nathan Dunfield  wrote:
>On Monday, February 19, 2024 at 3:08:54 PM UTC-6 John H Palmieri wrote, 
>responding to Dima:
>
>You said: "The difference between wheel packages vs pip packages is that 
>the latter don't require pre-fetched wheels, and absence of the need for 
>package (micro)management." The implication is that changing the package 
>management system is, maybe not part of this proposal, but a next step. In 
>other words, I'm getting this impression from your words, not by other 
>"certain parties."
>
>You said: "My proposal is in fact aimed at reducing the number of pinned 
>Sage dependecies, drastically." (You have made similar comments elsewhere 
>in this thread.) How does (1) accomplish this? Either I'm missing something 
>or you have not spelled everything out in your proposal.
>
>You said '"Allow" does not mean "Make all of the", it should be obvious.'
>
>"Allow" does not cause any changes to happen drastically. So what exactly 
>are you proposing to accomplish these drastic changes? If you have a 
>roadmap in mind, it would be helpful if you described it.
>
>
>My understanding is that allowing standard packages to be pip packages 
>could greatly reduce the number of pinned Sage dependencies for two reasons:
>
>1) a build-from-source or wheel package must explicitly pin its version, 
>but, more importantly,
>
>2) a pip package is allowed to install additional dependencies of PyPI that 
>are not recorded anywhere in the Sage repo.
>
>A simple example is pytest.  Here it is as an optional pip package:
>
>https://github.com/sagemath/sage/tree/develop/build/pkgs/pytest
>
>To be upgraded to a standard package, under the current policy would need 
>to be turned into a "wheel package" requires adding its dependencies like 
>so:
>
>https://github.com/sagemath/sage/pull/37301
>
>Here, pytest has just a few dependencies, but jupyterlab has more like 50 
>when you include dependencies of dependencies. 
>
>
>
>Personally, I think the current system of having everything pinned and 
>explicitly recorded is the right choice, being more stable in my experience 
>with other projects. 

The number of dependencies has grown to the point it has gotten too hard to 
maintain,  especially if one aims to support as many Python versions as we do.
These dependencies force one to have fragile, and temporary, version-dependent 
workarounds in the configuration. 
We don't have full-time (or even part-time) software engineers who can be 
tasked with such tedious stuff. Meanwhile maths bug pile up, but one has to do 
this ever growing maintenance...

That's why it's better to let go of as many explicit dependencies as possible 
now.
 


> In any event, switching to a pip package for e.g. 
>jupterlab doesn't affect the final size or complexity of Sage as installed, 
>just how many moving pieces there appear to be if you look in 
>"sage/build/pkgs".
>
>Best,
>
>Nathan
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/F329CEC3-622C-4F93-A676-5369F6C7B135%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-19 Thread Nathan Dunfield

On Monday, February 19, 2024 at 3:08:54 PM UTC-6 John H Palmieri wrote, 
responding to Dima:

You said: "The difference between wheel packages vs pip packages is that 
the latter don't require pre-fetched wheels, and absence of the need for 
package (micro)management." The implication is that changing the package 
management system is, maybe not part of this proposal, but a next step. In 
other words, I'm getting this impression from your words, not by other 
"certain parties."

You said: "My proposal is in fact aimed at reducing the number of pinned 
Sage dependecies, drastically." (You have made similar comments elsewhere 
in this thread.) How does (1) accomplish this? Either I'm missing something 
or you have not spelled everything out in your proposal.

You said '"Allow" does not mean "Make all of the", it should be obvious.'

"Allow" does not cause any changes to happen drastically. So what exactly 
are you proposing to accomplish these drastic changes? If you have a 
roadmap in mind, it would be helpful if you described it.


My understanding is that allowing standard packages to be pip packages 
could greatly reduce the number of pinned Sage dependencies for two reasons:

1) a build-from-source or wheel package must explicitly pin its version, 
but, more importantly,

2) a pip package is allowed to install additional dependencies of PyPI that 
are not recorded anywhere in the Sage repo.

A simple example is pytest.  Here it is as an optional pip package:

https://github.com/sagemath/sage/tree/develop/build/pkgs/pytest

To be upgraded to a standard package, under the current policy would need 
to be turned into a "wheel package" requires adding its dependencies like 
so:

https://github.com/sagemath/sage/pull/37301

Here, pytest has just a few dependencies, but jupyterlab has more like 50 
when you include dependencies of dependencies. 



Personally, I think the current system of having everything pinned and 
explicitly recorded is the right choice, being more stable in my experience 
with other projects.  In any event, switching to a pip package for e.g. 
jupterlab doesn't affect the final size or complexity of Sage as installed, 
just how many moving pieces there appear to be if you look in 
"sage/build/pkgs".

Best,

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/f81b8284-cb48-44fe-a3f7-158be2438335n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 19, 2024 at 2:41:11 PM UTC-8 Dima Pasechnik wrote:

On Mon, Feb 19, 2024 at 10:29 PM Matthias Koeppe  
wrote:

An option to "./configure" could work too, except that the "bootstrap" 
phase already downloads the "configure" tarball into that directory.


an option to ./bootstrap then would be logical


... except that "bootstrap" is invoked by "make configure", so it would 
have to be a "make" variable.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/bf4ca5ba-f880-4d30-9140-1cd3a02d8c7dn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

You're right, plus I was confusing "./bootstrap -d" (which is run by "make 
configure") with "./bootstrap -D" which forces the download.

On Monday, February 19, 2024 at 3:17:11 PM UTC-8 Matthias Koeppe wrote:

> On Monday, February 19, 2024 at 3:09:55 PM UTC-8 John H Palmieri wrote:
>
> By the way, I just cloned the Sage repo and ran "make configure", which 
> runs `./bootstrap`. The upstream directory is empty after that.
>
>
> You probably have autoconf/automake/... installed. In this case, it just 
> uses them to build the configure script. Downloading is a fallback that is 
> used when these "bootstrapping prerequisites" are not installed.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/ed16aa03-8b75-4dd1-bf74-567882d16646n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 19, 2024 at 3:09:55 PM UTC-8 John H Palmieri wrote:

By the way, I just cloned the Sage repo and ran "make configure", which 
runs `./bootstrap`. The upstream directory is empty after that.


You probably have autoconf/automake/... installed. In this case, it just 
uses them to build the configure script. Downloading is a fallback that is 
used when these "bootstrapping prerequisites" are not installed.


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/50bcb775-87a5-40dc-b0c5-e31000fc337dn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

If we keep a "configure" tarball in each separate Sage installation but 
they share the rest of "upstream", then we save a lot of space and a lot of 
downloading. A workflow like

% make configure
% configure --with-lots-of-options
% make

would be familiar and unchanged from the status quo when working with the 
git clone. Currently users don't have to run `./bootstrap` manually just to 
build Sage, and it would be nice to keep it this way.

By the way, I just cloned the Sage repo and ran "make configure", which 
runs `./bootstrap`. The upstream directory is empty after that. If it's 
getting used for temporary storage for this tarball, that can be done 
elsewhere (.upstream.d? config? some temporary directory?).

Another option for the location of a shared "upstream" would be Yet Another 
Environment Variable.

On Monday, February 19, 2024 at 2:41:11 PM UTC-8 Dima Pasechnik wrote:

> On Mon, Feb 19, 2024 at 10:29 PM Matthias Koeppe  
> wrote:
>
>> An option to "./configure" could work too, except that the "bootstrap" 
>> phase already downloads the "configure" tarball into that directory.
>
>
> an option to ./bootstrap then would be logical
>  
>
>>
>> Another possible direction: I've been thinking about creating a "./sage 
>> worktree" command, see https://github.com/sagemath/sage/issues/34744
>>
>>
>> On Monday, February 19, 2024 at 2:24:40 PM UTC-8 John H Palmieri wrote:
>>
>>> Regarding symlinking the upstream directory: instead or in addition, 
>>> what about an option to `./configure` for the location of that directory?
>>>
>>>
>>> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:
>>>
 Prompted by the discussion of space use on the *local machines* of 
 users and developers, I propose another item in addition to A and B:

 *C. Advertise use of "git worktree" and recommend symlinking the 
 "upstream" directory.* For testing a new release when you have an 
 existing clone of the repository, using "git clone" another time is 
 overkill as it creates another copy of the .git directory. And there is no 
 point in having multiple copies of the "upstream" directory, as the 
 filenames of the tarballs change whenever the contents change.

 On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote:

 This (A and B below) has the advantage of being quite explicit. The 
 original proposal

 1) allow standard packages to be pip packages 

 2) drop the contents of upstream/ from the Sage source tarballs. 

 sounds explicit, but the more the discussion goes on, the more I feel 
 that there are hidden pieces. Does the proposal also mean removing version 
 restrictions and all of the other claimed maintenance burden for various 
 components of Sage? 

 Regarding item (2): if I clone the github repository, there is no 
 upstream directory at the start, but after building Sage, it ends up being 
 almost as large as in the current tarballs. (This is on OS X with a lot of 
 homebrew packages installed.) So how much savings are we actually talking 
 about? (Maybe it's not savings for the end user that are important, so 
 *what* are we saving? Disk space on the mirrors?) Dima, can you please 
 provide data? If we convert (according to (1)) to pip packages, those 
 still 
 need to be downloaded, and while they may not end up in "upstream" — I 
 don't actually know how they work — don't they still take up disk space? 
 So 
 again, how much savings are we talking about? Please provide data.

 By the way, the git clone yields a package that is 616M on my machine. 
 A good chunk of that is the .git directory. Are you proposing that we do 
 not distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)

 Regarding item (1): can you provide a list of packages that would 
 become pip packages? Or describe how you would come up with a list?

 On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:

 I'll now offer:

 *Opinion 1. Nobody needs to care in the slightest what the size of that 
 release tarball is. *

 In any use cases with internet connectivity, people will be better off 
 by just cloning the git repo, not use the release tarball.

 If there are relevant use cases without internet connectivity (I have 
 no opinion to offer on this), then the release tarball has exactly the 
 right contents.

 *Proposed action items: *
 *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
 that "git clone" is described as the primary way to obtain the Sage 
 sources. That the big release tarball is available can be a footnote in 
 the 
 Installation Guide (
 https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)

 for the limited no-internet

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 19, 2024 at 10:29 PM Matthias Koeppe 
wrote:

> An option to "./configure" could work too, except that the "bootstrap"
> phase already downloads the "configure" tarball into that directory.


an option to ./bootstrap then would be logical


>
> Another possible direction: I've been thinking about creating a "./sage
> worktree" command, see https://github.com/sagemath/sage/issues/34744
>
>
> On Monday, February 19, 2024 at 2:24:40 PM UTC-8 John H Palmieri wrote:
>
>> Regarding symlinking the upstream directory: instead or in addition, what
>> about an option to `./configure` for the location of that directory?
>>
>>
>> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:
>>
>>> Prompted by the discussion of space use on the *local machines* of
>>> users and developers, I propose another item in addition to A and B:
>>>
>>> *C. Advertise use of "git worktree" and recommend symlinking the
>>> "upstream" directory.* For testing a new release when you have an
>>> existing clone of the repository, using "git clone" another time is
>>> overkill as it creates another copy of the .git directory. And there is no
>>> point in having multiple copies of the "upstream" directory, as the
>>> filenames of the tarballs change whenever the contents change.
>>>
>>> On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote:
>>>
>>> This (A and B below) has the advantage of being quite explicit. The
>>> original proposal
>>>
>>> 1) allow standard packages to be pip packages
>>>
>>> 2) drop the contents of upstream/ from the Sage source tarballs.
>>>
>>> sounds explicit, but the more the discussion goes on, the more I feel
>>> that there are hidden pieces. Does the proposal also mean removing version
>>> restrictions and all of the other claimed maintenance burden for various
>>> components of Sage?
>>>
>>> Regarding item (2): if I clone the github repository, there is no
>>> upstream directory at the start, but after building Sage, it ends up being
>>> almost as large as in the current tarballs. (This is on OS X with a lot of
>>> homebrew packages installed.) So how much savings are we actually talking
>>> about? (Maybe it's not savings for the end user that are important, so
>>> *what* are we saving? Disk space on the mirrors?) Dima, can you please
>>> provide data? If we convert (according to (1)) to pip packages, those still
>>> need to be downloaded, and while they may not end up in "upstream" — I
>>> don't actually know how they work — don't they still take up disk space? So
>>> again, how much savings are we talking about? Please provide data.
>>>
>>> By the way, the git clone yields a package that is 616M on my machine. A
>>> good chunk of that is the .git directory. Are you proposing that we do not
>>> distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)
>>>
>>> Regarding item (1): can you provide a list of packages that would become
>>> pip packages? Or describe how you would come up with a list?
>>>
>>>
>>> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>>>
>>> I'll now offer:
>>>
>>> *Opinion 1. Nobody needs to care in the slightest what the size of that
>>> release tarball is. *
>>>
>>> In any use cases with internet connectivity, people will be better off
>>> by just cloning the git repo, not use the release tarball.
>>>
>>> If there are relevant use cases without internet connectivity (I have no
>>> opinion to offer on this), then the release tarball has exactly the right
>>> contents.
>>>
>>> *Proposed action items: *
>>> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so
>>> that "git clone" is described as the primary way to obtain the Sage
>>> sources. That the big release tarball is available can be a footnote in the
>>> Installation Guide (
>>> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>>> for the limited no-internet connectivity use case.
>>>
>>> *B. *Likewise, get rid of all of these "Download Sage source code"
>>> pages (https://www.sagemath.org/download-source.html,
>>> https://www.sagemath.org/download-latest.html), mirror selection, etc.
>>> from the Sage website.
>>>
>>>
>>> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>>>
>>> Currently the standard packages cannot be pip packages, i.e. we must, in
>>> effect, vendor them. This entails an extra effort which is often not
>>> needed, in particular as we patch only very few Python packages.
>>> Pip packages are on the other hand installed straight from PyPI.
>>>
>>> Good examples of standard packages which can become pip ones are tox,
>>> pytest (not yet standard).
>>>
>>>
>>> The other difference is that by default these packages are not included
>>> in the Sage releases source tarball.
>>>
>>> Rather than adding them there I propose to split the upstream/* part of
>>> the tarball into something optional - which is represented by a list of
>>> files to download, and which

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

An option to "./configure" could work too, except that the "bootstrap" 
phase already downloads the "configure" tarball into that directory.

Another possible direction: I've been thinking about creating a "./sage 
worktree" command, see https://github.com/sagemath/sage/issues/34744


On Monday, February 19, 2024 at 2:24:40 PM UTC-8 John H Palmieri wrote:

> Regarding symlinking the upstream directory: instead or in addition, what 
> about an option to `./configure` for the location of that directory?
>
>
> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:
>
>> Prompted by the discussion of space use on the *local machines* of users 
>> and developers, I propose another item in addition to A and B:
>>
>> *C. Advertise use of "git worktree" and recommend symlinking the 
>> "upstream" directory.* For testing a new release when you have an 
>> existing clone of the repository, using "git clone" another time is 
>> overkill as it creates another copy of the .git directory. And there is no 
>> point in having multiple copies of the "upstream" directory, as the 
>> filenames of the tarballs change whenever the contents change.
>>
>> On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote:
>>
>> This (A and B below) has the advantage of being quite explicit. The 
>> original proposal
>>
>> 1) allow standard packages to be pip packages 
>>
>> 2) drop the contents of upstream/ from the Sage source tarballs. 
>>
>> sounds explicit, but the more the discussion goes on, the more I feel 
>> that there are hidden pieces. Does the proposal also mean removing version 
>> restrictions and all of the other claimed maintenance burden for various 
>> components of Sage? 
>>
>> Regarding item (2): if I clone the github repository, there is no 
>> upstream directory at the start, but after building Sage, it ends up being 
>> almost as large as in the current tarballs. (This is on OS X with a lot of 
>> homebrew packages installed.) So how much savings are we actually talking 
>> about? (Maybe it's not savings for the end user that are important, so 
>> *what* are we saving? Disk space on the mirrors?) Dima, can you please 
>> provide data? If we convert (according to (1)) to pip packages, those still 
>> need to be downloaded, and while they may not end up in "upstream" — I 
>> don't actually know how they work — don't they still take up disk space? So 
>> again, how much savings are we talking about? Please provide data.
>>
>> By the way, the git clone yields a package that is 616M on my machine. A 
>> good chunk of that is the .git directory. Are you proposing that we do not 
>> distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)
>>
>> Regarding item (1): can you provide a list of packages that would become 
>> pip packages? Or describe how you would come up with a list?
>>
>>
>> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>>
>> I'll now offer:
>>
>> *Opinion 1. Nobody needs to care in the slightest what the size of that 
>> release tarball is. *
>>
>> In any use cases with internet connectivity, people will be better off by 
>> just cloning the git repo, not use the release tarball.
>>
>> If there are relevant use cases without internet connectivity (I have no 
>> opinion to offer on this), then the release tarball has exactly the right 
>> contents.
>>
>> *Proposed action items: *
>> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
>> that "git clone" is described as the primary way to obtain the Sage 
>> sources. That the big release tarball is available can be a footnote in the 
>> Installation Guide (
>> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>>  
>> for the limited no-internet connectivity use case.
>>
>> *B. *Likewise, get rid of all of these "Download Sage source code" pages 
>> (https://www.sagemath.org/download-source.html, 
>> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
>> from the Sage website. 
>>
>>
>> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>>
>> Currently the standard packages cannot be pip packages, i.e. we must, in 
>> effect, vendor them. This entails an extra effort which is often not 
>> needed, in particular as we patch only very few Python packages. 
>> Pip packages are on the other hand installed straight from PyPI. 
>>
>> Good examples of standard packages which can become pip ones are tox, 
>> pytest (not yet standard). 
>>
>>
>> The other difference is that by default these packages are not included 
>> in the Sage releases source tarball. 
>>
>> Rather than adding them there I propose to split the upstream/* part of 
>> the tarball into something optional - which is represented by a list of 
>> files to download, and which is just not needed if you build while 
>> connected to the internet. 
>>
>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
>> tarball is

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

Regarding symlinking the upstream directory: instead or in addition, what 
about an option to `./configure` for the location of that directory?


On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote:

> Prompted by the discussion of space use on the *local machines* of users 
> and developers, I propose another item in addition to A and B:
>
> *C. Advertise use of "git worktree" and recommend symlinking the 
> "upstream" directory.* For testing a new release when you have an 
> existing clone of the repository, using "git clone" another time is 
> overkill as it creates another copy of the .git directory. And there is no 
> point in having multiple copies of the "upstream" directory, as the 
> filenames of the tarballs change whenever the contents change.
>
> On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote:
>
> This (A and B below) has the advantage of being quite explicit. The 
> original proposal
>
> 1) allow standard packages to be pip packages 
>
> 2) drop the contents of upstream/ from the Sage source tarballs. 
>
> sounds explicit, but the more the discussion goes on, the more I feel that 
> there are hidden pieces. Does the proposal also mean removing version 
> restrictions and all of the other claimed maintenance burden for various 
> components of Sage? 
>
> Regarding item (2): if I clone the github repository, there is no upstream 
> directory at the start, but after building Sage, it ends up being almost as 
> large as in the current tarballs. (This is on OS X with a lot of homebrew 
> packages installed.) So how much savings are we actually talking about? 
> (Maybe it's not savings for the end user that are important, so *what* are 
> we saving? Disk space on the mirrors?) Dima, can you please provide data? 
> If we convert (according to (1)) to pip packages, those still need to be 
> downloaded, and while they may not end up in "upstream" — I don't actually 
> know how they work — don't they still take up disk space? So again, how 
> much savings are we talking about? Please provide data.
>
> By the way, the git clone yields a package that is 616M on my machine. A 
> good chunk of that is the .git directory. Are you proposing that we do not 
> distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)
>
> Regarding item (1): can you provide a list of packages that would become 
> pip packages? Or describe how you would come up with a list?
>
>
> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>
> I'll now offer:
>
> *Opinion 1. Nobody needs to care in the slightest what the size of that 
> release tarball is. *
>
> In any use cases with internet connectivity, people will be better off by 
> just cloning the git repo, not use the release tarball.
>
> If there are relevant use cases without internet connectivity (I have no 
> opinion to offer on this), then the release tarball has exactly the right 
> contents.
>
> *Proposed action items: *
> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
> that "git clone" is described as the primary way to obtain the Sage 
> sources. That the big release tarball is available can be a footnote in the 
> Installation Guide (
> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>  
> for the limited no-internet connectivity use case.
>
> *B. *Likewise, get rid of all of these "Download Sage source code" pages (
> https://www.sagemath.org/download-source.html, 
> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
> from the Sage website. 
>
>
> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>
> Currently the standard packages cannot be pip packages, i.e. we must, in 
> effect, vendor them. This entails an extra effort which is often not 
> needed, in particular as we patch only very few Python packages. 
> Pip packages are on the other hand installed straight from PyPI. 
>
> Good examples of standard packages which can become pip ones are tox, 
> pytest (not yet standard). 
>
>
> The other difference is that by default these packages are not included in 
> the Sage releases source tarball. 
>
> Rather than adding them there I propose to split the upstream/* part of 
> the tarball into something optional - which is represented by a list of 
> files to download, and which is just not needed if you build while 
> connected to the internet. 
>
> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
> tarball is 1.3Gb, and without it is smaller than 0.25Gb. 
>
> Note that as William writes, the desire to have Sage buildable without an 
> internet connection was a requirement by a past Sage funder, gone about 10 
> years ago. Thus there's no longer an obligation to have this option. 
> I am not aware of a similar to Sage which provides tarballs allowing for 
> an offline build. 
>
> Thus, I would like to call a vote on these two topics: 
>
> 1) allow standard

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 19 February 2024 21:08:54 GMT, John H Palmieri  
wrote:
>
>
>On Monday, February 19, 2024 at 12:10:49 PM UTC-8 Dima Pasechnik wrote:
>
>On Mon, Feb 19, 2024 at 7:42 PM John H Palmieri wrote:
>
>This (A and B below) has the advantage of being quite explicit. The 
>original proposal
>
>1) allow standard packages to be pip packages 
>
>2) drop the contents of upstream/ from the Sage source tarballs. 
>
>sounds explicit, but the more the discussion goes on, the more I feel that 
>there are hidden pieces. Does the proposal also mean removing version 
>restrictions and all of the other claimed maintenance burden for various 
>components of Sage? 
>
>
>No, this is simply FUD spread by certain parties here. 
>
>
>You said: "The difference between wheel packages vs pip packages is that 
>the latter don't require pre-fetched wheels, and absence of the need for 
>package (micro)management." The implication is that changing the package 
>management system is, maybe not part of this proposal, but a next step. In 
>other words, I'm getting this impression from your words, not by other 
>"certain parties."
>
>You said: "My proposal is in fact aimed at reducing the number of pinned 
>Sage dependecies, drastically." (You have made similar comments elsewhere 
>in this thread.) How does (1) accomplish this? Either I'm missing something 
>or you have not spelled everything out in your proposal.
>
>You said '"Allow" does not mean "Make all of the", it should be obvious.'
>
>"Allow" does not cause any changes to happen drastically. So what exactly 
>are you proposing to accomplish these drastic changes? If you have a 
>roadmap in mind, it would be helpful if you described it.

I think we can drastically, by hundreds, reduce the number of Python packages 
we catalogue/vendor in build/pkgs/. Namely, these which are only used deep 
inside Jupyter, Sphinx, and Python build system, if we convert more standard 
Python packages into pip packages.

We can also hope to convert e.g. matplotlib, numpy, scipy, etc 
into pip packages, but this is not as obvious.

This is based on very limited experiments, though.
Anyhow, we have obvious, and safe, candidates for pip package conversion, such 
as pytest.

>
>
> 
>
>pytest is good example of package which can be elevated to standard, but 
>kept pip. In no place my
>proposal 1) demands anything done for all packages.
>
>
>
>Regarding item (2): if I clone the github repository, there is no upstream 
>directory at the start, but after building Sage, it ends up being almost as 
>large as in the current tarballs. (This is on OS X with a lot of homebrew 
>packages installed.) So how much savings are we actually talking about? 
>(Maybe it's not savings for the end user that are important, so *what* are 
>we saving? Disk space on the mirrors?) Dima, can you please provide data? 
>If we convert (according to (1)) to pip packages, those still need to be 
>downloaded, and while they may not end up in "upstream" — I don't actually 
>know how they work — don't they still take up disk space? So again, how 
>much savings are we talking about? Please provide data.
>
>
>I am talking about saving space on mirrors, and on bandwidth, by not 
>packaging "upstream/".
>(as I wrote: This is a huge saving on the tarball size: with upstream/* in, 
>Sage 10.2 tarball is 1.3Gb, and without it is smaller than 0.25Gb.)
>Local disk space nowadays is cheap, but space on mirrors, and bandwidth, 
>don't come free.
>(not everyone is on an unlimited internet)
>
>As well, if you don't wipe up your local upstream/, its contents can  be 
>reused.
>(typically not so many packages are updated with each release after all)
>
>However, as far as I can tell, by default pip package wheels are not stored 
>in upstream/.
>Perhaps there is an easy way to change this, I don't know.
> 
> 
>
>
>By the way, the git clone yields a package that is 616M on my machine. A 
>good chunk of that is the .git directory. Are you proposing that we do not 
>distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.) 
>
>
>Regarding item (1): can you provide a list of packages that would become 
>pip packages? Or describe how you would come up with a list?
>
>
>packages not involved in sagelib directly are good candidates, e.g. pytest, 
>tox.
>sphinx and jupyterlab are good candidates too, in my limited testing 
>experiments.
>
>Dima
>
>
>On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>
>I'll now offer:
>
>*Opinion 1. Nobody needs to care in the slightest what the size of that 
>release tarball is. *
>
>In any use cases with internet connectivity, people will be better off by 
>just cloning the git repo, not use the release tarball.
>
>If there are relevant use cases without internet connectivity (I have no 
>opinion to offer on this), then the release tarball has exactly the right 
>contents.
>
>*Proposed action items: *
>*A.* Change https://github.com/sagemath/sage/blob/develop/README.md so that 
>"git clone" is

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 19, 2024 at 12:10:49 PM UTC-8 Dima Pasechnik wrote:

On Mon, Feb 19, 2024 at 7:42 PM John H Palmieri  wrote:

If we convert (according to (1)) to pip packages, those still need to be 
downloaded, and while they may not end up in "upstream" — I don't actually 
know how they work — don't they still take up disk space? So again, how 
much savings are we talking about? Please provide data.


as far as I can tell, by default pip package wheels are not stored in 
upstream/.
Perhaps there is an easy way to change this, I don't know.


That's correct. pip packages are installed by directly calling "pip 
install".
If pip downloads a wheel file, it is not stored anywhere.
If pip downloads an sdist and builds a wheel file, then the built wheel is 
stored in pip's cache; see output from "./sage -pip cache info" and "./sage 
-pip cache list".


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/231f8ccb-b60f-49e4-b3cb-b1b88041d351n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size



On Monday, February 19, 2024 at 12:10:49 PM UTC-8 Dima Pasechnik wrote:

On Mon, Feb 19, 2024 at 7:42 PM John H Palmieri wrote:

This (A and B below) has the advantage of being quite explicit. The 
original proposal

1) allow standard packages to be pip packages 

2) drop the contents of upstream/ from the Sage source tarballs. 

sounds explicit, but the more the discussion goes on, the more I feel that 
there are hidden pieces. Does the proposal also mean removing version 
restrictions and all of the other claimed maintenance burden for various 
components of Sage? 


No, this is simply FUD spread by certain parties here. 


You said: "The difference between wheel packages vs pip packages is that 
the latter don't require pre-fetched wheels, and absence of the need for 
package (micro)management." The implication is that changing the package 
management system is, maybe not part of this proposal, but a next step. In 
other words, I'm getting this impression from your words, not by other 
"certain parties."

You said: "My proposal is in fact aimed at reducing the number of pinned 
Sage dependecies, drastically." (You have made similar comments elsewhere 
in this thread.) How does (1) accomplish this? Either I'm missing something 
or you have not spelled everything out in your proposal.

You said '"Allow" does not mean "Make all of the", it should be obvious.'

"Allow" does not cause any changes to happen drastically. So what exactly 
are you proposing to accomplish these drastic changes? If you have a 
roadmap in mind, it would be helpful if you described it.


 

pytest is good example of package which can be elevated to standard, but 
kept pip. In no place my
proposal 1) demands anything done for all packages.



Regarding item (2): if I clone the github repository, there is no upstream 
directory at the start, but after building Sage, it ends up being almost as 
large as in the current tarballs. (This is on OS X with a lot of homebrew 
packages installed.) So how much savings are we actually talking about? 
(Maybe it's not savings for the end user that are important, so *what* are 
we saving? Disk space on the mirrors?) Dima, can you please provide data? 
If we convert (according to (1)) to pip packages, those still need to be 
downloaded, and while they may not end up in "upstream" — I don't actually 
know how they work — don't they still take up disk space? So again, how 
much savings are we talking about? Please provide data.


I am talking about saving space on mirrors, and on bandwidth, by not 
packaging "upstream/".
(as I wrote: This is a huge saving on the tarball size: with upstream/* in, 
Sage 10.2 tarball is 1.3Gb, and without it is smaller than 0.25Gb.)
Local disk space nowadays is cheap, but space on mirrors, and bandwidth, 
don't come free.
(not everyone is on an unlimited internet)

As well, if you don't wipe up your local upstream/, its contents can  be 
reused.
(typically not so many packages are updated with each release after all)

However, as far as I can tell, by default pip package wheels are not stored 
in upstream/.
Perhaps there is an easy way to change this, I don't know.
 
 


By the way, the git clone yields a package that is 616M on my machine. A 
good chunk of that is the .git directory. Are you proposing that we do not 
distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.) 


Regarding item (1): can you provide a list of packages that would become 
pip packages? Or describe how you would come up with a list?


packages not involved in sagelib directly are good candidates, e.g. pytest, 
tox.
sphinx and jupyterlab are good candidates too, in my limited testing 
experiments.

Dima


On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:

I'll now offer:

*Opinion 1. Nobody needs to care in the slightest what the size of that 
release tarball is. *

In any use cases with internet connectivity, people will be better off by 
just cloning the git repo, not use the release tarball.

If there are relevant use cases without internet connectivity (I have no 
opinion to offer on this), then the release tarball has exactly the right 
contents.

*Proposed action items: *
*A.* Change https://github.com/sagemath/sage/blob/develop/README.md so that 
"git clone" is described as the primary way to obtain the Sage sources. 
That the big release tarball is available can be a footnote in the 
Installation Guide (
https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
 
for the limited no-internet connectivity use case.

*B. *Likewise, get rid of all of these "Download Sage source code" pages (
https://www.sagemath.org/download-source.html, 
https://www.sagemath.org/download-latest.html), mirror selection, etc. from 
the Sage website. 


On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:

Currently the standard packages cannot be pip packages, i.e. we must, in 
effect, vendor them. This entails an extra

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 19, 2024 at 7:42 PM John H Palmieri 
wrote:

> This (A and B below) has the advantage of being quite explicit. The
> original proposal
>
> 1) allow standard packages to be pip packages
>
> 2) drop the contents of upstream/ from the Sage source tarballs.
>
> sounds explicit, but the more the discussion goes on, the more I feel that
> there are hidden pieces. Does the proposal also mean removing version
> restrictions and all of the other claimed maintenance burden for various
> components of Sage?
>

No, this is simply FUD spread by certain parties here. "Allow" does not
mean "Make all of the", it should be
obvious.

pytest is good example of package which can be elevated to standard, but
kept pip. In no place my
proposal 1) demands anything done for all packages.



> Regarding item (2): if I clone the github repository, there is no upstream
> directory at the start, but after building Sage, it ends up being almost as
> large as in the current tarballs. (This is on OS X with a lot of homebrew
> packages installed.) So how much savings are we actually talking about?
> (Maybe it's not savings for the end user that are important, so *what* are
> we saving? Disk space on the mirrors?) Dima, can you please provide data?
> If we convert (according to (1)) to pip packages, those still need to be
> downloaded, and while they may not end up in "upstream" — I don't actually
> know how they work — don't they still take up disk space? So again, how
> much savings are we talking about? Please provide data.
>

I am talking about saving space on mirrors, and on bandwidth, by not
packaging "upstream/".
(as I wrote: This is a huge saving on the tarball size: with upstream/* in,
Sage 10.2 tarball is 1.3Gb, and without it is smaller than 0.25Gb.)
Local disk space nowadays is cheap, but space on mirrors, and bandwidth,
don't come free.
(not everyone is on an unlimited internet)

As well, if you don't wipe up your local upstream/, its contents can  be
reused.
(typically not so many packages are updated with each release after all)

However, as far as I can tell, by default pip package wheels are not stored
in upstream/.
Perhaps there is an easy way to change this, I don't know.



>
> By the way, the git clone yields a package that is 616M on my machine. A
> good chunk of that is the .git directory. Are you proposing that we do not
> distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)
>

> Regarding item (1): can you provide a list of packages that would become
> pip packages? Or describe how you would come up with a list?
>

packages not involved in sagelib directly are good candidates, e.g. pytest,
tox.
sphinx and jupyterlab are good candidates too, in my limited testing
experiments.

Dima


On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>
>> I'll now offer:
>>
>> *Opinion 1. Nobody needs to care in the slightest what the size of that
>> release tarball is. *
>>
>> In any use cases with internet connectivity, people will be better off by
>> just cloning the git repo, not use the release tarball.
>>
>> If there are relevant use cases without internet connectivity (I have no
>> opinion to offer on this), then the release tarball has exactly the right
>> contents.
>>
>> *Proposed action items: *
>> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so
>> that "git clone" is described as the primary way to obtain the Sage
>> sources. That the big release tarball is available can be a footnote in the
>> Installation Guide (
>> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>> for the limited no-internet connectivity use case.
>>
>> *B. *Likewise, get rid of all of these "Download Sage source code" pages
>> (https://www.sagemath.org/download-source.html,
>> https://www.sagemath.org/download-latest.html), mirror selection, etc.
>> from the Sage website.
>>
>>
>> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>>
>>> Currently the standard packages cannot be pip packages, i.e. we must, in
>>> effect, vendor them. This entails an extra effort which is often not
>>> needed, in particular as we patch only very few Python packages.
>>> Pip packages are on the other hand installed straight from PyPI.
>>>
>>> Good examples of standard packages which can become pip ones are tox,
>>> pytest (not yet standard).
>>>
>>>
>>> The other difference is that by default these packages are not included
>>> in the Sage releases source tarball.
>>>
>>> Rather than adding them there I propose to split the upstream/* part of
>>> the tarball into something optional - which is represented by a list of
>>> files to download, and which is just not needed if you build while
>>> connected to the internet.
>>>
>>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2
>>> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>>>
>>> Note that as William writes, the desire to have Sage buildable without

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

Prompted by the discussion of space use on the *local machines* of users 
and developers, I propose another item in addition to A and B:

*C. Advertise use of "git worktree" and recommend symlinking the "upstream" 
directory.* For testing a new release when you have an existing clone of 
the repository, using "git clone" another time is overkill as it creates 
another copy of the .git directory. And there is no point in having 
multiple copies of the "upstream" directory, as the filenames of the 
tarballs change whenever the contents change.

On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote:

This (A and B below) has the advantage of being quite explicit. The 
original proposal

1) allow standard packages to be pip packages 

2) drop the contents of upstream/ from the Sage source tarballs. 

sounds explicit, but the more the discussion goes on, the more I feel that 
there are hidden pieces. Does the proposal also mean removing version 
restrictions and all of the other claimed maintenance burden for various 
components of Sage? 

Regarding item (2): if I clone the github repository, there is no upstream 
directory at the start, but after building Sage, it ends up being almost as 
large as in the current tarballs. (This is on OS X with a lot of homebrew 
packages installed.) So how much savings are we actually talking about? 
(Maybe it's not savings for the end user that are important, so *what* are 
we saving? Disk space on the mirrors?) Dima, can you please provide data? 
If we convert (according to (1)) to pip packages, those still need to be 
downloaded, and while they may not end up in "upstream" — I don't actually 
know how they work — don't they still take up disk space? So again, how 
much savings are we talking about? Please provide data.

By the way, the git clone yields a package that is 616M on my machine. A 
good chunk of that is the .git directory. Are you proposing that we do not 
distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)

Regarding item (1): can you provide a list of packages that would become 
pip packages? Or describe how you would come up with a list?


On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:

I'll now offer:

*Opinion 1. Nobody needs to care in the slightest what the size of that 
release tarball is. *

In any use cases with internet connectivity, people will be better off by 
just cloning the git repo, not use the release tarball.

If there are relevant use cases without internet connectivity (I have no 
opinion to offer on this), then the release tarball has exactly the right 
contents.

*Proposed action items: *
*A.* Change https://github.com/sagemath/sage/blob/develop/README.md so that 
"git clone" is described as the primary way to obtain the Sage sources. 
That the big release tarball is available can be a footnote in the 
Installation Guide (
https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
 
for the limited no-internet connectivity use case.

*B. *Likewise, get rid of all of these "Download Sage source code" pages (
https://www.sagemath.org/download-source.html, 
https://www.sagemath.org/download-latest.html), mirror selection, etc. from 
the Sage website. 


On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:

Currently the standard packages cannot be pip packages, i.e. we must, in 
effect, vendor them. This entails an extra effort which is often not 
needed, in particular as we patch only very few Python packages. 
Pip packages are on the other hand installed straight from PyPI. 

Good examples of standard packages which can become pip ones are tox, 
pytest (not yet standard). 


The other difference is that by default these packages are not included in 
the Sage releases source tarball. 

Rather than adding them there I propose to split the upstream/* part of the 
tarball into something optional - which is represented by a list of files 
to download, and which is just not needed if you build while connected to 
the internet. 

This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
tarball is 1.3Gb, and without it is smaller than 0.25Gb. 

Note that as William writes, the desire to have Sage buildable without an 
internet connection was a requirement by a past Sage funder, gone about 10 
years ago. Thus there's no longer an obligation to have this option. 
I am not aware of a similar to Sage which provides tarballs allowing for an 
offline build. 

Thus, I would like to call a vote on these two topics: 

1) allow standard packages to be pip packages 

2) drop the contents of upstream/ from the Sage source tarballs. 


--- 
Dima 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

This (A and B below) has the advantage of being quite explicit. The 
original proposal

1) allow standard packages to be pip packages 

2) drop the contents of upstream/ from the Sage source tarballs. 

sounds explicit, but the more the discussion goes on, the more I feel that 
there are hidden pieces. Does the proposal also mean removing version 
restrictions and all of the other claimed maintenance burden for various 
components of Sage? 

Regarding item (2): if I clone the github repository, there is no upstream 
directory at the start, but after building Sage, it ends up being almost as 
large as in the current tarballs. (This is on OS X with a lot of homebrew 
packages installed.) So how much savings are we actually talking about? 
(Maybe it's not savings for the end user that are important, so *what* are 
we saving? Disk space on the mirrors?) Dima, can you please provide data? 
If we convert (according to (1)) to pip packages, those still need to be 
downloaded, and while they may not end up in "upstream" — I don't actually 
know how they work — don't they still take up disk space? So again, how 
much savings are we talking about? Please provide data.

By the way, the git clone yields a package that is 616M on my machine. A 
good chunk of that is the .git directory. Are you proposing that we do not 
distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.)

Regarding item (1): can you provide a list of packages that would become 
pip packages? Or describe how you would come up with a list?

On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:

> I'll now offer:
>
> *Opinion 1. Nobody needs to care in the slightest what the size of that 
> release tarball is. *
>
> In any use cases with internet connectivity, people will be better off by 
> just cloning the git repo, not use the release tarball.
>
> If there are relevant use cases without internet connectivity (I have no 
> opinion to offer on this), then the release tarball has exactly the right 
> contents.
>
> *Proposed action items: *
> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
> that "git clone" is described as the primary way to obtain the Sage 
> sources. That the big release tarball is available can be a footnote in the 
> Installation Guide (
> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>  
> for the limited no-internet connectivity use case.
>
> *B. *Likewise, get rid of all of these "Download Sage source code" pages (
> https://www.sagemath.org/download-source.html, 
> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
> from the Sage website. 
>
>
> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>
>> Currently the standard packages cannot be pip packages, i.e. we must, in 
>> effect, vendor them. This entails an extra effort which is often not 
>> needed, in particular as we patch only very few Python packages. 
>> Pip packages are on the other hand installed straight from PyPI. 
>>
>> Good examples of standard packages which can become pip ones are tox, 
>> pytest (not yet standard). 
>>
>>
>> The other difference is that by default these packages are not included 
>> in the Sage releases source tarball. 
>>
>> Rather than adding them there I propose to split the upstream/* part of 
>> the tarball into something optional - which is represented by a list of 
>> files to download, and which is just not needed if you build while 
>> connected to the internet. 
>>
>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
>> tarball is 1.3Gb, and without it is smaller than 0.25Gb. 
>>
>> Note that as William writes, the desire to have Sage buildable without an 
>> internet connection was a requirement by a past Sage funder, gone about 10 
>> years ago. Thus there's no longer an obligation to have this option. 
>> I am not aware of a similar to Sage which provides tarballs allowing for 
>> an offline build. 
>>
>> Thus, I would like to call a vote on these two topics: 
>>
>> 1) allow standard packages to be pip packages 
>>
>> 2) drop the contents of upstream/ from the Sage source tarballs. 
>>
>>
>> --- 
>> Dima 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/0a3c1c0b-ed3f-41eb-ab4a-5235432475dbn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 19, 2024 at 5:16 PM Matthias Koeppe 
wrote:

> On Monday, February 19, 2024 at 5:42:08 AM UTC-8 tobia...@gmx.de wrote:
>
> This discussion about the need to fix the version of pytest *and its
> runtime dependencies* is almost comical.
>
>
> No, you are in the wrong thread.
>
> This thread is about the general policy for standard packages, not about
> pytest.
>

Come on! Tobias is in the right thread.
pytest is a good example of standard package which can well be made a pip
package, that's why it's mentioned here.


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/e6477c9c-9a52-4b71-829d-b718cbbd5402n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq1xQ9fG%3DGLqvsWKTduQTiamQV2vPyvpphHXYxMFC5n21w%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 19, 2024 at 5:42:08 AM UTC-8 tobia...@gmx.de wrote:

This discussion about the need to fix the version of pytest *and its 
runtime dependencies* is almost comical.


No, you are in the wrong thread. 

This thread is about the general policy for standard packages, not about 
pytest. 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/e6477c9c-9a52-4b71-829d-b718cbbd5402n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-19 Thread 'tobia...@gmx.de' via sage-devel

This discussion about the need to fix the version of pytest *and its 
runtime dependencies* is almost comical. We are installing and running 
pytest successfully since 3 years without any version requirement via pip 
in ci and experienced zero issues. We are also not alone in that. For 
example, scipy also doesn't pin any pytest version 
(https://github.com/search?q=repo%3Ascipy%2Fscipy%20pytest%20lang%3Ayml=code).
 
Can you name a single project that needed to pin one of the runtime 
dependencies of pytest because of a bug they experienced?

On Monday, February 19, 2024 at 7:47:26 AM UTC+8 Matthias Koeppe wrote:

> On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:
>
> On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe  
> wrote:
>
> On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:
>
>   
>
> 2) The major improvement is that sagelib will be easier to install into an 
> existing venv, and that's a wish of quite a number of users. Much more 
> Pythonic, too.
>
>
> The pip-installability of sagelib has absolutely nothing to do with this 
> discussion.
>
>
> Of course it does, a lot. As having less deps pinning will model 
> installability and useability of sagelib in a "foreign" venv. At the moment 
> using sagelib in a foreign venv is complicated and error-prone, and 
> untested.
>
>
> The dependencies of sagelib are declared in the template 
> https://github.com/sagemath/sage/blob/develop/src/setup.cfg.m4#L13, which 
> is filled with the data from the "install-requires.txt" files (such as 
> https://github.com/sagemath/sage/blob/develop/build/pkgs/cysignals/install-requires.txt
> ).
> These files provide version ranges. They do NOT use the specific versions 
> pinned in the Sage distribution.
>
> I don't think it's necessary to comment much on the idea that we should 
> change our production environment (unpinning the pinned versions of the 
> Sage distribution) to make it "model installability and usability of 
> sagelib" in a less controlled environment.
>
> Testing sagelib (the sagemath-standard distribution package) in different 
> environments is not "complicated" -- because I've built the convenient 
> tooling for that. One does not have to make changes to the Sage 
> distribution for that. See 
> https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-standard/tox.ini#L18
>  
> for the different options. See also the developer's guide 
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging_sage_library#testing-the-distribution-in-virtual-environments-with-tox,
>  
> where all of this is documented.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/8c75300e-ae84-49ea-bd05-03443dd745ebn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe
wrote:

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

2) The major improvement is that sagelib will be easier to install into an
existing venv, and that's a wish of quite a number of users. Much more
Pythonic, too.

The pip-installability of sagelib has absolutely nothing to do with this
discussion.

Of course it does, a lot. As having less deps pinning will model
installability and useability of sagelib in a "foreign" venv. At the moment
using sagelib in a foreign venv is complicated and error-prone, and
untested.

The dependencies of sagelib are declared in the template
https://github.com/sagemath/sage/blob/develop/src/setup.cfg.m4#L13, which
is filled with the data from the "install-requires.txt" files (such
as
https://github.com/sagemath/sage/blob/develop/build/pkgs/cysignals/install-requires.txt).
These files provide version ranges. They do NOT use the specific versions
pinned in the Sage distribution.

I don't think it's necessary to comment much on the idea that we should
change our production environment (unpinning the pinned versions of the
Sage distribution) to make it "model installability and usability of
sagelib" in a less controlled environment.

Testing sagelib (the sagemath-standard distribution package) in different
environments is not "complicated" -- because I've built the convenient
tooling for that. One does not have to make changes to the Sage
distribution for that.
See
https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-standard/tox.ini#L18

for the different options. See also the developer's guide
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging_sage_library#testing-the-distribution-in-virtual-environments-with-tox,

where all of this is documented.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/85a4614c-e003-461d-91f3-d43f498bfcf1n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sun, Feb 18, 2024 at 6:57 PM Nathan Dunfield 
wrote:

> On Sunday, February 18, 2024 at 12:14:58 PM UTC-6 Dima Pasechnik wrote:
>
> Reading:
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
>
>
> The wheel you talk about is just another packaging of a source package,
> isn't it?
>
>
> No.
>
>
> Well, I might have used incorrect terminology, but our wheels are always
> "build from source" wheels.
>
>
> No, many pure Python wheels are simply downloaded in the their final form
> from PyPI, for example:
>
> https://github.com/sagemath/sage/tree/develop/build/pkgs/webcolors
> https://github.com/sagemath/sage/tree/develop/build/pkgs/jupyterlab
>

OK, I am sorry, I've hallucinated a non-existing format for distributing
Python packages.
(or perhaps I saw somewhere a package which has both a pure-Python wheel,
and platform-specific wheels
with optional  binary extensions)


Wheels are platform-specific, and this non-pure-python packages need to
have platform-specific wheels,
see  e.g.
https://pypi.org/project/matplotlib/#files

>
>
> Best,
>
> Nathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/9775f89c-4901-4a0f-a0e0-91b38c215f18n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2_vfJJ97m55SV-wsULkGLvbzki%2Bw56Jxw%2BkGbGJ1UWTQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-18 Thread Nathan Dunfield

On Sunday, February 18, 2024 at 12:14:58 PM UTC-6 Dima Pasechnik wrote:

Reading: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
 

The wheel you talk about is just another packaging of a source package, 
isn't it?


No.


Well, I might have used incorrect terminology, but our wheels are always 
"build from source" wheels.


No, many pure Python wheels are simply downloaded in the their final form 
from PyPI, for example:

https://github.com/sagemath/sage/tree/develop/build/pkgs/webcolors 
https://github.com/sagemath/sage/tree/develop/build/pkgs/jupyterlab

Best,

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9775f89c-4901-4a0f-a0e0-91b38c215f18n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sun, Feb 18, 2024 at 6:09 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:
>
> On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe 
> wrote:
>
> On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:
>
> 1) you can even just get a binary wheel of pytest installed - it is very
> fast, and robust.
>
>
> Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does.
> It installs pytest as a "wheel" package.
>
> there are wheels and wheels.
> Binary wheels don't need any building, Sage's wheel packages still do
> building from source - in case the package has C extensions, possibly
> cythonizing or running a similar built processs involving
> compilation/linking.
>
>
> No, Dima.
>
> Reading:
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
>
>
> The wheel you talk about is just another packaging of a source package,
> isn't it?
>
>
> No.
>

Well, I might have used incorrect terminology, but our wheels are always
"build from source" wheels.


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/e87b2425-e69c-42d6-8534-519e5eddaa63n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq00Et16AEfSdkNisi9tuGdps-cC2eTNvD-gxH-mXnovLQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sun, Feb 18, 2024 at 5:15 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 4:40:35 AM UTC-8 Dima Pasechnik wrote:
>
> On 17 February 2024 23:31:43 GMT, Matthias Koeppe 
> wrote:
> >On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote:
> >"wheel" type Sage packages, each of which is
> >primarily just the version number of a file on PyPI and its hash, is like
> a
> >"requirements.txt" file (or "conda-lock" file, for that matter) spread
> over
> >multiple directories. Personally, I don't view that as packaging a
> >dependency, but rather saving some metadata to aid
> >reliability/reproducibility.
> >
> >I'll note that in addition to aiding reliability/reproducibility, the
> >metadata in build/pkgs is also important for discoverability and
> >attribution.
>
> Merely pinning down the versions doesn't magically brings you
> reproducibility, unless you are also willing to pin down the OS version,
> and the hardware. [...]
>
>
> This is an instance of the "all or nothing" fallacy, and simultaneously a
> "straw man" fallacy (note that both Nathan and I said it "aids"
> reproducibility etc.)
>

I don't know where you see fallacies. You are just not willing to view the
existing Sage tooling under a critical loop,
IMHO.

>
> Besides, to create a pinning of all the versions of Python packages, just
> run the appropriate pip command, it will produce a full list of all the
> versions, ready to be used to reproduce the environment. No need to
> maintain these pinnings by hand.
>
> Yes, metadata is important, it's just make-work to maintain it manually.
> We don't need to carry out this make-work.
>
>
> Exactly, it matters what tooling is available. And every single little
> improvement of our tooling will likely have more value than this entire
> thread.
>

The value is in the eye of beholder, if you like.
The existing tooling is too rigid, my proposal aims to improve it by making
it more flexible.


>
> Yes, we don't want to maintain the metadata "manually".
> And we don't! We use the "sage --package" script (
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages
> )
>
> Yes, "pip freeze" will output the current versions of installed Python
> packages, to be saved as a requirements.txt file.
> What's missing is the tooling that would feed this version information
> back to our version files.
>

why do we need "our version files"? we need input to pip, and "pip freeze"
can generate such an input.



> That's wishlist item https://github.com/sagemath/sage/issues/37314,
> estimated effort: 15 minutes of work. Any takers?
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/16525787-4dec-4ded-b2c1-4019683a930an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0-tuzToFhvi7iNNxkgRTHYNSqSQrSewy%3DqfcHfVp%3DHwQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe  
wrote:

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

1) you can even just get a binary wheel of pytest installed - it is very 
fast, and robust.


Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does. It 
installs pytest as a "wheel" package. 

there are wheels and wheels.
Binary wheels don't need any building, Sage's wheel packages still do 
building from source - in case the package has C extensions, possibly 
cythonizing or running a similar built processs involving 
compilation/linking.


No, Dima.

Reading: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
 

The wheel you talk about is just another packaging of a source package, 
isn't it?


No.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/e87b2425-e69c-42d6-8534-519e5eddaa63n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:
>
> 1) you can even just get a binary wheel of pytest installed - it is very
> fast, and robust.
>
>
> Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does.
> It installs pytest as a "wheel" package.
>
there are wheels and wheels.
Binary wheels don't need any building, Sage's wheel packages still do
building from source - in case the package has C extensions, possibly
cythonizing or running a similar built processs involving
compilation/linking.
The wheel you talk about is just another packaging of a source package,
isn't it?

Just the other day we saw how one can very well install pyscipopt, an
(optional) Sage package,
from a binary wheel, without building (cause Sage's Cython is too new for
pyscipopt, one can't in present build it from source).
https://groups.google.com/g/sage-devel/c/htVeuOH_3t8/m/mIjBH7aRAgAJ
Doing the same with e.g. scipy will shave off quite a bit  of build time
(by for this scipy needs to become a pip package).



> Whether you install a package from an sdist or from a wheel, you still
> have the same runtime dependencies ("install-requires").
> What goes away when you use a wheel is only the build-time dependencies
> ("build-system requires").
>

>
> 2) The major improvement is that sagelib will be easier to install into an
> existing venv, and that's a wish of quite a number of users. Much more
> Pythonic, too.
>
>
> The pip-installability of sagelib has absolutely nothing to do with this
> discussion.
>

Of course it does, a lot. As having less deps pinning will model
installability and useability of sagelib in a "foreign" venv. At the moment
using sagelib in a foreign venv is complicated and error-prone, and
untested.

And I should also say that less deps will make it easier for downstreams to
package sage/sagelib.



>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/3299fe86-fa67-4830-9f3c-1386235c37c8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0jkCCKzxhFGFLLsZYacUPzZU%2BhvkPUroKKc2XNBHTbeg%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

1) you can even just get a binary wheel of pytest installed - it is very 
fast, and robust.


Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does. It 
installs pytest as a "wheel" package.

Whether you install a package from an sdist or from a wheel, you still have 
the same runtime dependencies ("install-requires"). 
What goes away when you use a wheel is only the build-time dependencies 
("build-system requires").
  

2) The major improvement is that sagelib will be easier to install into an 
existing venv, and that's a wish of quite a number of users. Much more 
Pythonic, too.


The pip-installability of sagelib has absolutely nothing to do with this 
discussion.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3299fe86-fa67-4830-9f3c-1386235c37c8n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 18, 2024 at 4:40:35 AM UTC-8 Dima Pasechnik wrote:

On 17 February 2024 23:31:43 GMT, Matthias Koeppe  
wrote: 
>On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote: 
>"wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like 
a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread 
over 
>multiple directories. Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility. 
> 
>I'll note that in addition to aiding reliability/reproducibility, the 
>metadata in build/pkgs is also important for discoverability and 
>attribution. 

Merely pinning down the versions doesn't magically brings you 
reproducibility, unless you are also willing to pin down the OS version, 
and the hardware. [...]

This is an instance of the "all or nothing" fallacy, and simultaneously a 
"straw man" fallacy (note that both Nathan and I said it "aids" 
reproducibility etc.)

Besides, to create a pinning of all the versions of Python packages, just 
run the appropriate pip command, it will produce a full list of all the 
versions, ready to be used to reproduce the environment. No need to 
maintain these pinnings by hand. 

Yes, metadata is important, it's just make-work to maintain it manually. We 
don't need to carry out this make-work.

Exactly, it matters what tooling is available. And every single little 
improvement of our tooling will likely have more value than this entire 
thread.

Yes, we don't want to maintain the metadata "manually". 
And we don't! We use the "sage --package" script 
(https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages)

Yes, "pip freeze" will output the current versions of installed Python 
packages, to be saved as a requirements.txt file. 
What's missing is the tooling that would feed this version information back 
to our version files. That's wishlist 
item https://github.com/sagemath/sage/issues/37314, estimated effort: 15 
minutes of work. Any takers?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/16525787-4dec-4ded-b2c1-4019683a930an%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On 18 February 2024 15:51:27 GMT, Nathan Dunfield  wrote:
>On Sunday, February 18, 2024 at 6:26:28 AM UTC-6 Dima Pasechnik wrote:
>
>I cannot imagine CI breaking down by, say, pytest.
>
>
>I can definitely see that happening, and indeed it seems to have done so 
>for other projects:
>
>https://github.com/pytest-dev/pytest/issues/9765
>https://github.com/pytest-dev/pytest/issues/11983
> 

Well, yes, a major pytest version jump might be an issue. We know this well, so 
what?

>
>Besides, you can pin down or limit the version of a pip package, just as 
>well. E.g. pin down the version of Cython. But leave its dependencies out 
>of Sage, as much as possible.
>
>
>Leaving out dependencies doesn't eliminate complexity, it just hides it.  

Why would two dependencies of pytest* and pytest* only need to enter Sage 
codebase? No, there is no point in it, to the contrary it is more make-work, 
more resources, longer build times, etc.

>That doesn't seem like an improvement to me. 

well:

1) you can even just get a binary wheel of pytest installed - it is very fast, 
and robust. Just like you do with pytest being optional. There is no need for 
any extra deps getting in.

2) The major improvement is that sagelib will be easier to install into an 
existing venv, and that's a wish of quite a number of users. Much more 
Pythonic, too.

>  Plus, some packages are 
>dependencies of multiple other packages, and pip's ability to find a 
>version of said dependency that all other packages will tolerate is not the 
>greatest in my experience.

with Sage unnecessary pinning down a lot of package versions (and simply 
vendoring them,) you cannot expect pip to do too well. We just have to let go 
of many dependencies which are, needlessly, part of Sage, then pip will be able 
to do the job well.

Dima

>
>Best,
>
>Nathan
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9B2C761F-C865-4332-8106-6EA364A89FBD%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-18 Thread Nathan Dunfield

On Sunday, February 18, 2024 at 6:26:28 AM UTC-6 Dima Pasechnik wrote:

I cannot imagine CI breaking down by, say, pytest.


I can definitely see that happening, and indeed it seems to have done so 
for other projects:

https://github.com/pytest-dev/pytest/issues/9765
https://github.com/pytest-dev/pytest/issues/11983
 

Besides, you can pin down or limit the version of a pip package, just as 
well. E.g. pin down the version of Cython. But leave its dependencies out 
of Sage, as much as possible.


Leaving out dependencies doesn't eliminate complexity, it just hides it.  
That doesn't seem like an improvement to me.  Plus, some packages are 
dependencies of multiple other packages, and pip's ability to find a 
version of said dependency that all other packages will tolerate is not the 
greatest in my experience.

Best,

Nathan



-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/73328e5d-6e5a-4245-823f-957a3b2ef230n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 17 February 2024 23:31:43 GMT, Matthias Koeppe  
wrote:
>On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote:
>
>It seems to me that the "wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread over 
>multiple directories.  Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility.
>
>
>I agree with this viewpoint.
>I'll note that in addition to aiding reliability/reproducibility, the 
>metadata in build/pkgs is also important for discoverability and 
>attribution.

Merely pinning down the versions doesn't magically brings you reproducibility, 
unless you are also willing to pin down the OS version, and the hardware. It's 
often just doesn't make sence.
E.g. try to build Sage 8 or 9 on a modern machine with a recent OS. Chances are 
it won't work, despite you knowing all the versions.

Besides, to create a pinning of all the versions of Python packages, just run 
the appropriate pip command, it will produce a full list of all the versions, 
ready to be used to reproduce the environment. No need to maintain these 
pinnings by hand.

Yes, metadata is important, it's just make-work to maintain it manually. We 
don't need to carry out this make-work.


 
>
>Of course one could point out that our format is relatively verbose, as it 
>is spread over several plain-text files -- for simplicity of access with 
>shell scripts. Our format dates back to at least 2015, see the sage-devel 
>thread "Is there an online index of the standard packages shipped in Sage?" 
>(https://groups.google.com/g/sage-devel/c/aEmUmFOwJYQ/m/4pmmYrt3nXQJ) Other 
>distributions such as Pyodide 
>(https://github.com/pyodide/pyodide/tree/main/packages) use contemporary 
>structured datafile formats such as yaml instead. But it is probably safe 
>to say that most Sage users and even most Sage developers do not look at 
>the contents of the build/pkgs/ directory. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/14A14401-9869-4DA1-9C53-7C37A0FD5466%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 17 February 2024 23:04:49 GMT, Nathan Dunfield  wrote:
>On Saturday, February 17, 2024 at 1:13:33 PM UTC-6 Dima Pasechnik wrote:
>
>My proposal is in fact aimed at reducing the number of pinned Sage 
>dependecies, drastically. 
>
>Because most of them are either dependencies of Jupyterlab, or of Sphinx, 
>or of Python build system, and none of the them should be Sage's concern to 
>package, with all their dependencies.
>
>
>In my experience, it's particularly important to pin build dependencies.  
>Most of the "out of the blue" CI failures we've seen with "snappy" have 
>been caused by new versions of build dependencies, especially Cython. 


My proposal does not say that we must convert
all the standard python packages into pip packages.


I cannot imagine CI breaking down by, say, pytest.
Surely once every few years one might get a serious incompatibility, but not in 
10% of CI runs, or in even 1%.

Besides, you can pin down or limit the version of a pip package, just as well. 
E.g. pin down the version of Cython. But leave its dependencies out of Sage, as 
much as possible.




>  (I 
>see that Cython was also one of the motivations for the "conda-lock" scheme 
>of https://github.com/sagemath/sage/pull/35986 )
>
>If you itch to pack the said dependencies, please do it in a separate 
>repo/PyPI package, which can be consumed by sagelib to get the desired 
>pinned dependencies (and test all this in the existing CI, why not?) 
>But stop tying them up with sagelib - which in effect forces people 
>interested in sagelib to slave away on packaging 300 dependencies, most of 
>which aren't even tested by CI in any way, besides building.
>
>
>It seems to me that the "wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread over 
>multiple directories.  Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility.
>
>Best,
>
>Nathan
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/4F1F879D-76CB-4B59-A579-46868E4C8F23%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Matthias Koeppe

On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote:

It seems to me that the "wheel" type Sage packages, each of which is
primarily just the version number of a file on PyPI and its hash, is like a
"requirements.txt" file (or "conda-lock" file, for that matter) spread over
multiple directories. Personally, I don't view that as packaging a
dependency, but rather saving some metadata to aid
reliability/reproducibility.

I agree with this viewpoint.
I'll note that in addition to aiding reliability/reproducibility, the
metadata in build/pkgs is also important for discoverability and
attribution.

Of course one could point out that our format is relatively verbose, as it
is spread over several plain-text files -- for simplicity of access with
shell scripts. Our format dates back to at least 2015, see the sage-devel
thread "Is there an online index of the standard packages shipped in Sage?"
(https://groups.google.com/g/sage-devel/c/aEmUmFOwJYQ/m/4pmmYrt3nXQJ) Other
distributions such as Pyodide
(https://github.com/pyodide/pyodide/tree/main/packages) use contemporary
structured datafile formats such as yaml instead. But it is probably safe
to say that most Sage users and even most Sage developers do not look at
the contents of the build/pkgs/ directory.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/df37948c-b153-4bbd-97a4-0a532b1778fen%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Nathan Dunfield

On Saturday, February 17, 2024 at 1:13:33 PM UTC-6 Dima Pasechnik wrote:

My proposal is in fact aimed at reducing the number of pinned Sage
dependecies, drastically.

Because most of them are either dependencies of Jupyterlab, or of Sphinx,
or of Python build system, and none of the them should be Sage's concern to
package, with all their dependencies.

In my experience, it's particularly important to pin build dependencies.
Most of the "out of the blue" CI failures we've seen with "snappy" have
been caused by new versions of build dependencies, especially Cython. (I
see that Cython was also one of the motivations for the "conda-lock" scheme
of https://github.com/sagemath/sage/pull/35986 )

If you itch to pack the said dependencies, please do it in a separate
repo/PyPI package, which can be consumed by sagelib to get the desired
pinned dependencies (and test all this in the existing CI, why not?)
But stop tying them up with sagelib - which in effect forces people
interested in sagelib to slave away on packaging 300 dependencies, most of
which aren't even tested by CI in any way, besides building.

Best,

Nathan

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/c1d9fe0f-c196-4fe3-b11f-2d8673f782a7n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Matthias Koeppe

On Saturday, February 17, 2024 at 11:13:33 AM UTC-8 Dima Pasechnik wrote:

On 17 February 2024 17:16:07 GMT, Matthias Koeppe
wrote:
>I share the same concern based on the amplification of the failure
>probability, due to the large number of dependencies in Sage.

My proposal is in fact aimed at reducing the number of pinned Sage
dependecies, drastically.

As you seem to be responding to what I wrote there, I have to point out
that it is the *unpinning*, not the *pinning*, that is the concern for
instability that Nathan and I share.

Because most of them are either dependencies of Jupyterlab, or of Sphinx,
or of Python build system, and none of the them should be Sage's concern to
package, with all their dependencies.

Please liberate sagelib from the cabal of the ftontend, etc.

Sagemath is not a disto - no sane distro puts everything in one flat
directory structure.
Sagemath is an insane pile of needlessly vendored packages.

Wow, there's so much to unpack here...

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/cf23d2b4-be5a-45f4-ac6d-f1922ce140f3n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 17 February 2024 17:16:07 GMT, Matthias Koeppe  
wrote:
>On Saturday, February 17, 2024 at 7:06:27 AM UTC-8 Nathan Dunfield wrote:
>
>On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote:
>
>If one does not care about the use case without internet access, then it's 
>just the following:
>- Pinning, as you mentioned (see also 
>https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above, 
>where I discussed some details of this, including risks of leaving packages 
>unpinned)
>- Dependencies: "pip" packages can pull some of their build-time and 
>run-time dependencies directly from PyPI, without us mirroring these 
>dependencies in SageMath metadata. That's a mild convenience for 
>developers, of importance if one wants to leave the version range wide 
>open; but also has risks of instability.
>
>
>Matthias, thanks for the clarification.  I think pinning the version of a 
>"standard" package, including all its dependencies and down to the minor 
>release, is likely the best approach.  Based on my experience with snappy 
>[1], not pinning things will result in CI runs failing "out of the blue" 
>because one of the dependencies got updated.  With a small project like 
>snappy, this is pretty occasional and serves as a way to flag issues with 
>new upstream releases, but with Sage my guess is that such failures would 
>be frequent.   Suppose that each time the CI runs on a new PR, there's a 
>10% chance of it failing because some completely unrelated dependency 
>shifted; that would be a major annoyance to seasoned Sage developers and 
>very discouraging to newcomers.
>
>
>Thanks a lot, Nathan, for this point.
>I share the same concern based on the amplification of the failure 
>probability, due to the large number of dependencies in Sage.


My proposal is in fact aimed at reducing the number of pinned Sage dependecies, 
drastically.

Because most of them are either dependencies of Jupyterlab, or of Sphinx, or of 
Python build system, and none of the them should be Sage's concern to package, 
with all their dependencies.

If you itch to pack the said dependencies, please do it in a separate repo/PyPI 
package, which can be consumed by sagelib to get the desired pinned 
dependencies (and test all this in the existing CI, why not?)
But stop tying them up with sagelib - which in effect forces people interested 
in sagelib to slave away on packaging 300 dependencies, most of which aren't 
even tested by CI in any way, besides building.

Please liberate sagelib from the cabal of the ftontend, etc.

Sagemath is not a disto - no sane distro puts everything in one flat directory 
structure.
Sagemath is an insane pile of needlessly vendored packages.









>
>I'll note that recently we have merged PR 
>https://github.com/sagemath/sage/pull/35986 by Tobias Diez, which 
>implements full version pinning for our conda-forge installation methods. 
>See 
>https://github.com/sagemath/sage/blob/develop/src/environment-dev-3.11-macos-arm64.yml#L329
> 
>-- pytest is fully pinned down to the build hash. This change was motivated 
>specifically by the fast changes of conda-forge (as a rolling 
>distribution), which was one major cause for the instability of the CI 
>Conda workflow.
>The tooling for this is https://conda.github.io/conda-lock/ and Tobias's 
>script 
>https://github.com/sagemath/sage/blob/develop/.github/workflows/conda-lock-update.py
>
>The proposed policy change to allow standard packages to be "pip" packages 
>would run in the opposite direction.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/F661E14B-700E-4DCF-959A-DED3EFFB5898%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 17 February 2024 15:01:14 GMT, Kwankyu Lee  wrote:
>
>
>there are ways to use pip without internet, with the necessary wheels 
>pre-fetched. 
>That's what Sage does with wheel packages.
>
>
>Yes. This is a sage package of source type "wheel", as Matthias explained.
> 
>
>The difference between wheel packages vs pip packages is that the latter 
>don't require pre-fetched wheels, and absence of the need for package 
>(micro)management.
>
>
>So your proposal is to let a standard' package to be installed by pip via 
>internet. Hence your proposal suggests to break the rule that 
>sage-the-distribution can be installed without internet connection. If so, 
>breaking the rule (that is, assuming internet connection at install time) 
>is more substantial content of your proposal.


Once again: my proposal does not address the question how one can pack 
everything for an offline install.

We suspect the demand for this feature does not exist (while the requirements 
for it are gone long ago), but there are options to create a fully offline 
installer readily available.

Dima 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3FDD96B0-313D-4334-A751-E5AA8AA402E8%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Matthias Koeppe

On Saturday, February 17, 2024 at 7:06:27 AM UTC-8 Nathan Dunfield wrote:

On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote:

If one does not care about the use case without internet access, then it's
just the following:
- Pinning, as you mentioned (see also
https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above,
where I discussed some details of this, including risks of leaving packages
unpinned)
- Dependencies: "pip" packages can pull some of their build-time and
run-time dependencies directly from PyPI, without us mirroring these
dependencies in SageMath metadata. That's a mild convenience for
developers, of importance if one wants to leave the version range wide
open; but also has risks of instability.

Matthias, thanks for the clarification. I think pinning the version of a
"standard" package, including all its dependencies and down to the minor
release, is likely the best approach. Based on my experience with snappy
[1], not pinning things will result in CI runs failing "out of the blue"
because one of the dependencies got updated. With a small project like
snappy, this is pretty occasional and serves as a way to flag issues with
new upstream releases, but with Sage my guess is that such failures would
be frequent. Suppose that each time the CI runs on a new PR, there's a
10% chance of it failing because some completely unrelated dependency
shifted; that would be a major annoyance to seasoned Sage developers and
very discouraging to newcomers.

Thanks a lot, Nathan, for this point.
I share the same concern based on the amplification of the failure
probability, due to the large number of dependencies in Sage.

I'll note that recently we have merged PR
https://github.com/sagemath/sage/pull/35986 by Tobias Diez, which
implements full version pinning for our conda-forge installation methods.
See
https://github.com/sagemath/sage/blob/develop/src/environment-dev-3.11-macos-arm64.yml#L329

-- pytest is fully pinned down to the build hash. This change was motivated
specifically by the fast changes of conda-forge (as a rolling
distribution), which was one major cause for the instability of the CI
Conda workflow.
The tooling for this is https://conda.github.io/conda-lock/ and Tobias's
script
https://github.com/sagemath/sage/blob/develop/.github/workflows/conda-lock-update.py

The proposed policy change to allow standard packages to be "pip" packages
would run in the opposite direction.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/c01e6162-99b2-44f0-9173-10ee199bd536n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Nathan Dunfield

On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote:

So I am against allowing standard packages to be "pip" packages --- the
"wheel" approach seems like the right tradeoff here as (a) the final
package is sourced straight off PyPI so we don't have to build it (b) the
result is completely stable.

Best,

Nathan

[1] https://snappy.computop.org

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/5b0a8e66-3b2c-48df-8287-258d8252decdn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-17 Thread Kwankyu Lee



there are ways to use pip without internet, with the necessary wheels 
pre-fetched. 
That's what Sage does with wheel packages.


Yes. This is a sage package of source type "wheel", as Matthias explained.
 

The difference between wheel packages vs pip packages is that the latter 
don't require pre-fetched wheels, and absence of the need for package 
(micro)management.


So your proposal is to let a standard' package to be installed by pip via 
internet. Hence your proposal suggests to break the rule that 
sage-the-distribution can be installed without internet connection. If so, 
breaking the rule (that is, assuming internet connection at install time) 
is more substantial content of your proposal.


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6250e304-eb52-452b-9fc6-03121cdd57f7n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size




On 17 February 2024 02:26:32 GMT, Kwankyu Lee  wrote:
>
>
> 
>
>By default the package content would be fetched, as pip does,
>
>
>Not just as pip does, but by actually calling "pip" to contact PyPI.
> 
>
>and that would mean the default configuration for sage would require 
>internet at install time.
>
>
>That's right.
>
>
>Then Dima's proposal implies assuming internet at install time. Right? 
>
>I asked the same question before. But Dima denied it. Whence I got 
>confused...


there are ways to use pip without internet, with the necessary wheels 
pre-fetched.
That's what Sage does with wheel packages. The difference between wheel 
packages vs pip packages is that the latter don't require pre-fetched wheels, 
and absence of the need for package (micro)management.

>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/E2A6AEBB-9885-48A9-BD83-AA5C913BACC4%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size



On 17 February 2024 02:26:32 GMT, Kwankyu Lee  wrote:
>
>
> 
>
>By default the package content would be fetched, as pip does,
>
>
>Not just as pip does, but by actually calling "pip" to contact PyPI.
> 
>
>and that would mean the default configuration for sage would require 
>internet at install time.
>
>
>That's right.
>
>
>Then Dima's proposal implies assuming internet at install time. Right? 
>
>I asked the same question before. But Dima denied it. Whence I got 
>confused...
>
>-- 
>You received this message because you are subscribed to the Google Groups 
>"sage-devel" group.
>To unsubscribe from this group and stop receiving emails from it, send an 
>email to sage-devel+unsubscr...@googlegroups.com.
>To view this discussion on the web visit 
>https://groups.google.com/d/msgid/sage-devel/63525bee-fdc8-4b8f-918e-ae475deae35en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6CDD08D5-FB9F-4382-B8D0-9877A4C8A72F%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Friday, February 16, 2024 at 8:44:06 PM UTC-8 Nathan Dunfield wrote:

Dima mentioned "tox" [1] as an example of a "standard" package that would 
benefit from being switched to a "pip" package.  The "tox" package is pure 
python, so could also made a "wheel" package, which are already allowed for 
standard package, for example [2]. 


Yes, in fact, tox and its dependencies have already been "wheel" packages, 
see https://github.com/sagemath/sage/blob/develop/build/pkgs/tox/checksums.ini, 
since https://github.com/sagemath/sage/pull/36176 (Sep 2023). 
 
I have been switching many packages from "normal" to "wheel", which has 
reduced the complexity of the Sage distribution, as wheel packages have no 
installation scripts -- and also no build dependencies. The latter was 
crucial -- as we decided to install JupyterLab components from the 
pre-built wheels, which eliminated the complexity of Javascript build 
infrastructure from the Sage distribution, 
https://github.com/sagemath/sage/pull/36129

For wheel packages, it's all just metadata and the copied-over package 
README (which we need for building our reference manual).

I'm having difficultly understanding the practical differences between a 
"wheel" package and a "pip" packages in this setting. 


With "wheel", the wheel is downloaded from PyPI and put in upstream/ by 
various GH actions and put in the sage tarball and copied over to the sage 
mirrors, whereas with "pip" it is only downloaded by pip itself when an 
end-user builds Sage.  But in terms of developer effort, the only 
difference I see between "wheel" and "pip" is that the former has a few 
extra checksums, compare [2] and [3].

 

  What distinctions am I missing?  Is it that a "wheel" must be pinned to a 
specific release on PyPI whereas "pip" can specify a range?


If one does not care about the use case without internet access, then it's 
just the following:
- Pinning, as you mentioned (see also 
https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above, 
where I discussed some details of this, including risks of leaving packages 
unpinned)
- Dependencies: "pip" packages can pull some of their build-time and 
run-time dependencies directly from PyPI, without us mirroring these 
dependencies in SageMath metadata. That's a mild convenience for 
developers, of importance if one wants to leave the version range wide 
open; but also has risks of instability.

Obviously, what is costly or inconvenient for developers depends a lot on 
the tooling that is available. I can elaborate on this if there's interest.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/c91c1112-0405-41f8-8cdb-71b1942a5c8dn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-16 Thread Nathan Dunfield

Dima mentioned "tox" [1] as an example of a "standard" package that would 
benefit from being switched to a "pip" package.  The "tox" package is pure 
python, so could also made a "wheel" package, which are already allowed for 
standard package, for example [2].  I'm having difficultly understanding 
the practical differences between a "wheel" package and a "pip" packages in 
this setting.  With "wheel", the wheel is downloaded from PyPI and put in 
upstream/ by various GH actions and put in the sage tarball and copied over 
to the sage mirrors, whereas with "pip" it is only downloaded by pip itself 
when an end-user builds Sage.  But in terms of developer effort, the only 
difference I see between "wheel" and "pip" is that the former has a few 
extra checksums, compare [2] and [3].  What distinctions am I missing?  Is 
it that a "wheel" must be pinned to a specific release on PyPI whereas 
"pip" can specify a range?

Best,

Nathan

[1] https://github.com/sagemath/sage/tree/develop/build/pkgs/tox
[2] https://github.com/sagemath/sage/tree/develop/build/pkgs/webcolors
[3] https://github.com/sagemath/sage/tree/develop/build/pkgs/snappy

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/30bb7f74-e4f0-4122-84a7-7e7fb3db1f99n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Friday, February 16, 2024 at 6:26:32 PM UTC-8 Kwankyu Lee wrote:

By default the package content would be fetched, as pip does,

Not just as pip does, but by actually calling "pip" to contact PyPI.

and that would mean the default configuration for sage would require
internet at install time.

That's right.

Then Dima's proposal implies assuming internet at install time. Right?

Yes.

But one can make "pip" work with some local directory for the packages it
considers instead of using PyPI over the Internet:
We can use "pip install --no-index --find-links=/SOME/LOCAL/DIRECTORY ...".
See https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages

As all pip options can also be provided systematically via environment
variables, we can also set "PIP_NO_INDEX=true" and
"PIP_FIND_LINKS=/SOME/LOCAL/DIRECTORY" for the same effect. Then one does
not need to change the invocations of pip.

In fact, we already do exactly this in the Sage distribution for a slightly
different purpose, namely when we build "normal" Python packages and
"script" Python packages (= packages whose source trees are part of the
repository, such as
https://github.com/sagemath/sage/tree/develop/pkgs/sagemath-bliss), see
https://github.com/sagemath/sage/blob/develop/build/pkgs/sagemath_objects/spkg-install.in#L3

We do this because, following modern Python build practices, we build most
packages with "build isolation". The build-time prerequisites are not
accessed from the normal Sage venv but are specifically installed in a
temporary environment just for the build of the specific package. The
prerequisites are installed from wheel files in venv/var/lib/sage/wheels/;
this directory is referred to by the variable $SAGE_SPKG_WHEELS.

(Where do the wheel files in venv/var/lib/sage/wheels/ come from? Either
(1) we have built them ahead of time and stored them there; or (2) they are
platform-independent wheels and we have found them in the directory
upstream/, downloaded them from GH Release assets, downloaded them from
Sage mirrors, or the upstream_url (= PyPI).)

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/069c83d2-b54b-4026-96cc-8c01be13289dn%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-16 Thread Kwankyu Lee



 

By default the package content would be fetched, as pip does,


Not just as pip does, but by actually calling "pip" to contact PyPI.
 

and that would mean the default configuration for sage would require 
internet at install time.


That's right.


Then Dima's proposal implies assuming internet at install time. Right? 

I asked the same question before. But Dima denied it. Whence I got 
confused...

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/63525bee-fdc8-4b8f-918e-ae475deae35en%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Friday, February 16, 2024 at 3:57:06 PM UTC-8 Nils Bruin wrote:

As far as I understand, the proposal is to allow sage "packages" to be 
closer to more standard python prerequisites by letting them be resolved by 
pip packages.


No, we already have such Sage packages: This is just one of the 4 existing 
package "source types" (
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-types)
 
- "normal", "wheel", "pip", "script".

Most of our Python packages come from PyPI already. The difference is 
really (1) when we determine the version to be installed, and (2) if and 
how we distribute the tarball.

- "normal" packages are built from an sdist (tarball) retrieved from PyPI. 
- The version is set in the file package-version.txt, and the PyPI download 
URL ("upstream_url") and checksums are recorded in checksums.ini; 
see https://github.com/sagemath/sage/tree/develop/build/pkgs/numpy for an 
example. 
- The release manager's scripts download the package from the upstream_url 
and put them on the Sage mirrors. 
- If the package is standard, it is also included in the big release 
tarball. 
- If the package is standard and a stable release is being made, a GH 
Actions workflow also uploads the tarball as a Release Asset to GitHub (see 
https://github.com/sagemath/sage/releases/tag/10.2). 
- When users install Sage from git, any normal package is first attempted 
to retrieve from the GitHub Release Assets, then from Sage mirrors, then 
from the upstream_url. 
- When users install Sage from the big release tarball, standard normal 
packages have their sdists already in upstream, and only 
optional/experimental normal packages need to be retrieved.

What Dima proposes here is to allow _standard_ Sage packages to be of 
"source type" "pip". 
 

By default the package content would be fetched, as pip does,


Not just as pip does, but by actually calling "pip" to contact PyPI.
 

and that would mean the default configuration for sage would require 
internet at install time.


That's right.
 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/ac7bdb04-3a96-45a0-9c4c-0694000e4655n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-16 Thread Nils Bruin

As far as I understand, the proposal is to allow sage "packages" to be
closer to more standard python prerequisites by letting them be resolved by
pip packages. By default the package content would be fetched, as pip does,
and that would mean the default configuration for sage would require
internet at install time.

I also understand that there are people who are concerned that this may not
reflect all scenarios where people want to install sagemath and they would
prefer if there is a clear method to install sagemath from a well-defined
set of archives (one big one?) that need to be transferred to the target
machine, after which the install can proceed without internet access.

Searching for "pip without internet" gives various hits. One that at least
superficially looks like a reasonable starting point:

https://thilinamad.medium.com/install-python-packages-via-pip-without-an-internet-connection-b3dee83b4c2d

but there are also stackoverflow answers that look relevant.

It looks like, with a bit of work, pip can be convinced to look at local
files to satisfy prerequisites and packages. Hence, if we keep that in mind
it seems to me that having an archive of "pip packages" would be doable if
we ensure pip gets used in a way that makes it easy to reconfigure the
place to look for prereqs. Then it may be fairly easy to make an offline
installable version of sagemath, either by packing a big tarball that
includes the pip content or by making that available in a separate ball,
with an easy switch (or perhaps we can configure pip to first look locally
and then try the internet? Or the other way around? that it could
transition gracefully between different ways of satisfying requirements).

So, perhaps we can have our pips and networkless installs too?

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/db063109-24ab-4b0b-abf1-eac3d1a8003en%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 12, 2024 at 4:05:05 PM UTC-8 Dima Pasechnik wrote:

On Mon, Feb 12, 2024 at 11:52 PM Matthias Koeppe 
 wrote: 
> If there are relevant use cases without internet connectivity (I have no 
opinion to offer on this), then the release tarball has exactly the right 
contents. 

This won't be true any more if we allow standard packages to be pip 
packages.


That's correct.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/bb4bed3c-d053-489b-91e7-5bff1c81cc1an%40googlegroups.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-12 Thread 'tobia...@gmx.de' via sage-devel

+1 for both proposals.

Via "pip download" (https://pip.pypa.io/en/stable/cli/pip_download/) it is 
easy to resolve and download all pip packages on a system with internet 
connection, and then later on the target system install it without the need 
for internet.

On Tuesday, February 13, 2024 at 9:42:07 AM UTC+7 Matthias Koeppe wrote:

> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:
>
> In any use cases with internet connectivity, people will be better off by 
> just cloning the git repo, not use the release tarball.
>
> If there are relevant use cases without internet connectivity (I have no 
> opinion to offer on this), then the release tarball has exactly the right 
> contents.
>
> *Proposed action items: *
> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so 
> that "git clone" is described as the primary way to obtain the Sage 
> sources. That the big release tarball is available can be a footnote in the 
> Installation Guide (
> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>  
> for the limited no-internet connectivity use case.
>
>
> That's now https://github.com/sagemath/sage/pull/37309 (needs review)
>  
>
> *B. *Likewise, get rid of all of these "Download Sage source code" pages (
> https://www.sagemath.org/download-source.html, 
> https://www.sagemath.org/download-latest.html), mirror selection, etc. 
> from the Sage website. 
>
>
> That's now https://github.com/sagemath/website/pull/466
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/1b90f070-914f-45a6-bd4f-7a207b05448en%40googlegroups.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote:

In any use cases with internet connectivity, people will be better off by
just cloning the git repo, not use the release tarball.

If there are relevant use cases without internet connectivity (I have no
opinion to offer on this), then the release tarball has exactly the right
contents.

*Proposed action items: *
*A.* Change https://github.com/sagemath/sage/blob/develop/README.md so that
"git clone" is described as the primary way to obtain the Sage sources.
That the big release tarball is available can be a footnote in the
Installation Guide (
https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)

for the limited no-internet connectivity use case.

That's now https://github.com/sagemath/sage/pull/37309 (needs review)

*B. *Likewise, get rid of all of these "Download Sage source code" pages (
https://www.sagemath.org/download-source.html,
https://www.sagemath.org/download-latest.html), mirror selection, etc. from
the Sage website.

That's now https://github.com/sagemath/website/pull/466

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/1cb9073b-3459-4a36-ae09-f32fba92ffb6n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 11:52 PM Matthias Koeppe
 wrote:
>
> I'll now offer:
>
> Opinion 1. Nobody needs to care in the slightest what the size of that 
> release tarball is.

Not quite true. E.g. the  mirrors are not of infinite size, e.g. some
projects (symengine is an example, IIRC) on PyPI get constrained that
way.

>
> In any use cases with internet connectivity, people will be better off by 
> just cloning the git repo, not use the release tarball.
>
> If there are relevant use cases without internet connectivity (I have no 
> opinion to offer on this), then the release tarball has exactly the right 
> contents.

This won't be true any more if we allow standard packages to be pip packages.

>
> Proposed action items:
> A. Change https://github.com/sagemath/sage/blob/develop/README.md so that 
> "git clone" is described as the primary way to obtain the Sage sources. That 
> the big release tarball is available can be a footnote in the Installation 
> Guide 
> (https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)
>  for the limited no-internet connectivity use case.
>
> B. Likewise, get rid of all of these "Download Sage source code" pages 
> (https://www.sagemath.org/download-source.html, 
> https://www.sagemath.org/download-latest.html), mirror selection, etc. from 
> the Sage website.
>
>
> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>>
>> Currently the standard packages cannot be pip packages, i.e. we must, in 
>> effect, vendor them. This entails an extra effort which is often not needed, 
>> in particular as we patch only very few Python packages.
>> Pip packages are on the other hand installed straight from PyPI.
>>
>> Good examples of standard packages which can become pip ones are tox, pytest 
>> (not yet standard).
>>
>>
>> The other difference is that by default these packages are not included in 
>> the Sage releases source tarball.
>>
>> Rather than adding them there I propose to split the upstream/* part of the 
>> tarball into something optional - which is represented by a list of files to 
>> download, and which is just not needed if you build while connected to the 
>> internet.
>>
>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
>> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>>
>> Note that as William writes, the desire to have Sage buildable without an 
>> internet connection was a requirement by a past Sage funder, gone about 10 
>> years ago. Thus there's no longer an obligation to have this option.
>> I am not aware of a similar to Sage which provides tarballs allowing for an 
>> offline build.
>>
>> Thus, I would like to call a vote on these two topics:
>>
>> 1) allow standard packages to be pip packages
>>
>> 2) drop the contents of upstream/ from the Sage source tarballs.
>>
>>
>> ---
>> Dima
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/f926e074-9803-4335-b128-29398c460b0en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0QmtZ5hY39%3DSmfmdtP97wqnQDBstRq7iAd_EwVJHHxXw%40mail.gmail.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

I'll now offer:

*Opinion 1. Nobody needs to care in the slightest what the size of that 
release tarball is. *

In any use cases with internet connectivity, people will be better off by 
just cloning the git repo, not use the release tarball.

If there are relevant use cases without internet connectivity (I have no 
opinion to offer on this), then the release tarball has exactly the right 
contents.

*Proposed action items: *
*A.* Change https://github.com/sagemath/sage/blob/develop/README.md so that 
"git clone" is described as the primary way to obtain the Sage sources. 
That the big release tarball is available can be a footnote in the 
Installation Guide 
(https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps)

for the limited no-internet connectivity use case.

*B. *Likewise, get rid of all of these "Download Sage source code" pages 
(https://www.sagemath.org/download-source.html, 
https://www.sagemath.org/download-latest.html), mirror selection, etc. from 
the Sage website. 

On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:

> Currently the standard packages cannot be pip packages, i.e. we must, in 
> effect, vendor them. This entails an extra effort which is often not 
> needed, in particular as we patch only very few Python packages.
> Pip packages are on the other hand installed straight from PyPI.
>
> Good examples of standard packages which can become pip ones are tox, 
> pytest (not yet standard).
>
>
> The other difference is that by default these packages are not included in 
> the Sage releases source tarball.
>
> Rather than adding them there I propose to split the upstream/* part of 
> the tarball into something optional - which is represented by a list of 
> files to download, and which is just not needed if you build while 
> connected to the internet.
>
> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>
> Note that as William writes, the desire to have Sage buildable without an 
> internet connection was a requirement by a past Sage funder, gone about 10 
> years ago. Thus there's no longer an obligation to have this option.
> I am not aware of a similar to Sage which provides tarballs allowing for 
> an offline build.
>
> Thus, I would like to call a vote on these two topics:
>
> 1) allow standard packages to be pip packages 
>
> 2) drop the contents of upstream/ from the Sage source tarballs.
>
>
> ---
> Dima
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/f926e074-9803-4335-b128-29398c460b0en%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-12 Thread John H Palmieri

What does this (a discussion of how Sage specifies version restrictions) 
have to do with the proposal? If it's relevant, that was not clear in the 
original proposal, so please clarify. It sounds like you might be proposing 
removing version checks on many of the packages Sage uses, or at least 
that's a conclusion I might draw from your critique of the amount of 
maintenance for Sage packages. Or maybe you are proposing redesigning the 
version specification system? In any case, it wasn't stated as part of the 
original proposal, so I don't know what was intended. If it is not relevant 
to the proposal, let's drop this part of the discussion.

I would also suggest dropping the question of whether we're "vendoring." 
The proposal clearly says that we should stop distributing the tarballs in 
the upstream directory, so whatever we call it, that part is clear.

(Maybe by "vendoring" you meant the combination of including the tarballs 
and the maintenance on the allowed versions, or maybe just including the 
tarballs, or maybe something else. The word "vendoring" does not seem to be 
helpful, so instead spelling out exactly what's meant for Sage could be 
helpful, at least if you meant more than just removing "upstream".)

On Monday, February 12, 2024 at 3:07:38 PM UTC-8 Dima Pasechnik wrote:

> On Mon, Feb 12, 2024 at 10:01 PM Matthias Koeppe
>  wrote:
> >
> > On Monday, February 12, 2024 at 10:49:04 AM UTC-8 Dima Pasechnik wrote:
> >
> > requirements.txt might as well specify the range, and this is used too 
> e.g.
> >
> > build/pkgs/phitigra/requirements.txt has
> > phitigra>=0.2.6
> >
> >
> > Yes, as I said in 
> https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ, 
> ""Pip" packages can either be pinned to a specific version, or set 
> acceptable version ranges, or be entirely unconstrained. This is set in the 
> file requirements.txt in the package directory."
> >
> > So this is all [...] confusing
> >
> >
> > That's why I'm taking the time to explain it clearly for the benefit of 
> everyone.
>
> I am sorry: I claimed that Sage has about 5 different ways to
> specify/restrict versions of its packages,
> and this makes it hugely confusing.
> You disagreed, but now you say that it needs an explanation.
>
> What really needs an explanation is how we ever went this far on a
> garden path. :-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/b89c5e20-0fb0-4c92-8333-4f1766a2bfa1n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 10:01 PM Matthias Koeppe
 wrote:
>
> On Monday, February 12, 2024 at 10:49:04 AM UTC-8 Dima Pasechnik wrote:
>
> requirements.txt might as well specify the range, and this is used too e.g.
>
> build/pkgs/phitigra/requirements.txt has
> phitigra>=0.2.6
>
>
> Yes, as I said in 
> https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ, ""Pip" 
> packages can either be pinned to a specific version, or set acceptable 
> version ranges, or be entirely unconstrained. This is set in the file 
> requirements.txt in the package directory."
>
> So this is all [...] confusing
>
>
> That's why I'm taking the time to explain it clearly for the benefit of 
> everyone.

I am sorry: I claimed that Sage has about 5 different ways to
specify/restrict versions of its packages,
and this makes it hugely confusing.
You disagreed, but now you say that it needs an explanation.

What really needs an explanation is how we ever went this far on a
garden path. :-)

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2CVh1bbe2EishyAkGUtn%3DEMgcOyonoLUTG2sAYSXZwtQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 10:11 PM Matthias Koeppe
 wrote:
>
> On Monday, February 12, 2024 at 10:49:04 AM UTC-8 Dima Pasechnik wrote:
>
> > 2. Also the large Sage source tarball does not "vendor". It is a shipment 
> > of a distribution. Distributions don't "vendor". It's the job of a 
> > distribution to ship its components.
> This is not correct. Sage is not a distribution
>
>
> Let's not do "Sage-the-distribution is not a distirbution" again. 
> https://groups.google.com/g/sage-devel/c/3Zoq0CNE1hE/m/tPgFOpHWBwAJ (2023).

I never agreed with William on this one (Sage is too narrow in scope
and incomplete to be a distribution),
Anaconda calls itself "distribution", Sage is quite far from
Anaconda's functionality.

Anyway, William concludes with "I hope soon Sage isn't a distribution,
but right now it still is. "
Do you also hope for the latter?

Anyhow, it's just fuzzy terminology, as well as just what exactly "to
vendor" means.
With the definition of "to vendor" I provided then you got to agree
that we vendor a lot of things.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq29Gfat-GOf0UiVrFiOmxRrvo_pp3VmjFrQm0FK8xGfuQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 12, 2024 at 10:49:04 AM UTC-8 Dima Pasechnik wrote:

> 2. Also the large Sage source tarball does not "vendor". It is a shipment 
of a distribution. Distributions don't "vendor". It's the job of a 
distribution to ship its components.
This is not correct. Sage is not a distribution


Let's not do "Sage-the-distribution is not a distirbution" again. 
https://groups.google.com/g/sage-devel/c/3Zoq0CNE1hE/m/tPgFOpHWBwAJ (2023).

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/93e3aedc-ad58-4048-8833-f38238d6b874n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 12, 2024 at 10:49:04 AM UTC-8 Dima Pasechnik wrote:

requirements.txt might as well specify the range, and this is used too e.g.

build/pkgs/phitigra/requirements.txt has
phitigra>=0.2.6


Yes, as I said 
in https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ, 
""Pip" packages can either be pinned to a specific version, or set 
acceptable version ranges, or be entirely unconstrained. This is set in the 
file requirements.txt in the package directory."

So this is all [...] confusing


That's why I'm taking the time to explain it clearly for the benefit of 
everyone.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/c798c68c-66a5-4d6a-a3e9-2f1ff0c6f1c9n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 6:02 PM Matthias Koeppe 
wrote:
>
> On Monday, February 12, 2024 at 3:18:05 AM UTC-8 Dima Pasechnik wrote:
>
> > Pinning packages to a set of tested working versions is a standard
practice, and as a matter of fact part of best practices to achieve
stability in various deployment situations, reproducibility, etc.
> >
> > In the Python world, such pinning is done using requirements.txt,
Pipfile.lock, and environment.yml files.
> > In the Sage distribution, we pin using package-version.txt and tiny
requirements.txt files.
>
> as well as install-requires.txt and spkg-configure.m4 - they also in
> some cases pin versions, strictly,or not.
>
>
> These files serve a different purpose. They declare acceptable version
ranges.

requirements.txt might as well specify the range, and this is used too e.g.

build/pkgs/phitigra/requirements.txt has

phitigra>=0.2.6

So this is all blurred and confusing

> In pure Python packages, this exists as well, as you know.
> It is done in pyproject.toml "dependencies" (previously setup.cfg/py
"install-requires").
>
> Talking about these here is a distraction that does not serve the
discussion of this topic.
>
> Now, at last, tell us what makes Sage so special that we must vendor
> sphinx and jupyter [...]
>
>
> Note that I have not expressed much of an opinion yet on your proposal.
> We'll get there.
>
> But as I have pointed out several times previously, you are using the
word "vendoring" in a polemic and idiosyncratic way, which does not serve
the discussion. More below.
>
> > A question to ask is what tooling is available to update the version
pins, and what the cost of using the tools is. For a typical upgrade, by
improving our tooling, we have reduced the work to just typing "./sage
-package update-latest sphinx --commit". In the Sphinx upgrade,
https://github.com/sagemath/sage/pull/37129/files (needs review), I ended
up updating 25 packages, so I had to use a command like this 25 times. It's
repetitive, maybe it takes 20 minutes total, but it's not remotely
something that I would use the phrase "Sage has shot itself in the foot"
for.
>
> The whole thing of a zillion vendored packages [...]
>
>
> 1. Sage does not "vendor". What is in build/pkgs is _metadata_. It's just
text. Sage _pins_ versions of packages, so there is information on the
version.

of course, I never said that metadata is vendoring, it's certainly not, and
this is a deviation from the topic.

>
> 2. Also the large Sage source tarball does not "vendor". It is a shipment
of a distribution. Distributions don't "vendor". It's the job of a
distribution to ship its components.
This is not correct. Sage is not a distribution, and  I am using the verb
as described here:  https://en.wiktionary.org/wiki/vendor#Verb

*vendor* (third-person singular simple present *vendors*, present
participle *vendoring*, simple past and past participle *vendored*)

1. (transitive
, software
engineering ) To
bundle third-party  dependencies
 with the source code
 for one's own program.
*  I distributed my application with a vendored copy of
Perl so that it wouldn't use the system copies of Perl where it is
installed.*

   1. (transitive
   , software
   engineering ) As
   the software vendor, to bundle one's own, possibly modified version of
   dependencies  with a
   standard program. *Strawberry Perl contains vendored copies of some CPAN
   modules, designed to allow them to run on Windows.*

According to this definition, everything in upstream/ is vendored (except
our own packages, like configure.)


>
> --
> You received this message because you are subscribed to the Google Groups
"sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/e2fd4a63-c029-4c1a-92eb-4a81c3ac6a16n%40googlegroups.com
.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq1U%3DjrpU3Yn4K%2BX9kpm7p%3DENgLqch2z%2BH3ABPaK8fBU6g%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Monday, February 12, 2024 at 3:18:05 AM UTC-8 Dima Pasechnik wrote:

> Pinning packages to a set of tested working versions is a standard
practice, and as a matter of fact part of best practices to achieve
stability in various deployment situations, reproducibility, etc.
>
> In the Python world, such pinning is done using requirements.txt,
Pipfile.lock, and environment.yml files.
> In the Sage distribution, we pin using package-version.txt and tiny
requirements.txt files.

as well as install-requires.txt and spkg-configure.m4 - they also in
some cases pin versions, strictly,or not.

These files serve a different purpose. They declare acceptable version
ranges.
In pure Python packages, this exists as well, as you know.
It is done in pyproject.toml "dependencies" (previously setup.cfg/py
"install-requires").

Talking about these here is a distraction that does not serve the
discussion of this topic.

Now, at last, tell us what makes Sage so special that we must vendor
sphinx and jupyter [...]

Note that I have not expressed much of an opinion yet on your proposal.
We'll get there.

But as I have pointed out several times previously, you are using the word
"vendoring" in a polemic and idiosyncratic way, which does not serve the
discussion. More below.

> A question to ask is what tooling is available to update the version
pins, and what the cost of using the tools is. For a typical upgrade, by
improving our tooling, we have reduced the work to just typing "./sage
-package update-latest sphinx --commit". In the Sphinx upgrade,
https://github.com/sagemath/sage/pull/37129/files (needs review), I ended
up updating 25 packages, so I had to use a command like this 25 times. It's
repetitive, maybe it takes 20 minutes total, but it's not remotely
something that I would use the phrase "Sage has shot itself in the foot"
for.

The whole thing of a zillion vendored packages [...]

1. Sage does not "vendor". What is in build/pkgs is _metadata_. It's just
text. Sage _pins_ versions of packages, so there is information on the
version.

2. Also the large Sage source tarball does not "vendor". It is a shipment
of a distribution. Distributions don't "vendor". It's the job of a
distribution to ship its components.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/e2fd4a63-c029-4c1a-92eb-4a81c3ac6a16n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 12:34 PM kcrisman  wrote:
>
> As part of this thread, I'd again ask for a discussion of the following 
> situation I asked in the other thread.  Dima had some interesting points 
> about a less-vendored approach saving disk space etc., but it would be 
> helpful to have input from people who have had to install Sage in these kinds 
> of situations en masse.  Separately, I'm also wondering about the Windows 
> situation since much of the world, for better or worse, is not on Linux.

On Windows, once you have WSL 2 up and running in a default way
(something that it's very common to have, and it's beyond the scope of
Sage how to have it on in detail)
you basically are on a recent Ubuntu (assessed via a weird interface, but OK).


>
> "At least in the not too distant past there have been situations where the 
> non-requirement of internet connectivity alleviated issues of limited 
> internet accessibility in a given locale, limited download speeds, limited 
> grid electricity, etc.   This policy just as much affects those situations, 
> and perhaps some people who have installed Sage in such environments 
> (including Sage Days and other events) might want to weigh in on that, and 
> whether such situations still obtain (as I personally assume they must 
> certainly do).  I figure three-letter agencies have people with the skills to 
> get around not using pip install, but if your downloads are over a mobile 
> network (or, for that matter, Project Kuiper or Starlink or whatever), you 
> might still want to download Sage - especially now that we don't have binary 
> installs "provided"."
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/1c003881-c1c5-4d5a-8fd3-fb78d46263f7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0_-sKSB5k4yrO-E%3DqAuOrAR6kW2HBcE30jsUGisH4ojQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-12 Thread kcrisman

As part of this thread, I'd again ask for a discussion of the following 
situation I asked in the other thread.  Dima had some interesting points 
about a less-vendored approach saving disk space etc., but it would be 
helpful to have input from people who have had to install Sage in these 
kinds of situations en masse.  Separately, I'm also wondering about the 
Windows situation since much of the world, for better or worse, is not on 
Linux.

"At least in the not too distant past there have been situations where the 
non-requirement of internet connectivity alleviated issues of limited 
internet accessibility in a given locale, limited download speeds, limited 
grid electricity, etc.   This policy just as much affects those situations, 
and perhaps some people who have installed Sage in such environments 
(including Sage Days and other events) might want to weigh in on that, and 
whether such situations still obtain (as I personally assume they must 
certainly do).  I figure three-letter agencies have people with the skills 
to get around not using pip install, but if your downloads are over a 
mobile network (or, for that matter, Project Kuiper or Starlink or 
whatever), you might still want to download Sage - especially now that we 
don't have binary installs "provided"."

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/1c003881-c1c5-4d5a-8fd3-fb78d46263f7n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Mon, Feb 12, 2024 at 12:57 AM Matthias Koeppe
 wrote:
>
> On Sunday, February 11, 2024 at 3:34:46 PM UTC-8 Dima Pasechnik wrote:
>
> On 11 February 2024 22:47:24 GMT, Matthias Koeppe  
> wrote:
> >On Sunday, February 11, 2024 at 1:46:40 PM UTC-8 Matthias Koeppe wrote:
> >
> >I'll make an attempt to quantify this cost
> >
> >Here's an illustration of the workflow for making python_build a standard
> >"wheel" package, as proposed in
> >https://groups.google.com/g/sage-devel/c/MIU-xo9b7pc:
>
> What you outlined is the initial one-time cost.
>
>
> That's correct, that's what I did in that post.
>
>
> There is also a cost of maintenance, which eventually gets bigger than the 
> initial cost: the thing gets outdated, its dependencies get outdated, this 
> all requires updates, tests, conflict resolutions ---something that you get 
> largely for free if you let go of the package dependency micromanagement, 
> relying instead on the Python universe out there to do the job.
>
>
> That's where a possible sleight of hand happens.
> Let's please do this discussion at normal speed, giving the audience a chance 
> to observe the facts and form their opinion.
>
> Pinning packages to a set of tested working versions is a standard practice, 
> and as a matter of fact part of best practices to achieve stability in 
> various deployment situations, reproducibility, etc.
>
> In the Python world, such pinning is done using requirements.txt, 
> Pipfile.lock, and environment.yml files.
> In the Sage distribution, we pin using package-version.txt and tiny 
> requirements.txt files.

as well as install-requires.txt and spkg-configure.m4 - they also in
some cases pin versions, strictly,or not.
Now you can lament about the lack of more developers joining the
project... (they come, they see the insanity of controlling versions
in 5 different somewhat incompatible ways, they leave).

>
> When updating the pins, testing is always necessary; it does not come for 
> free. Yes, we have our automatic tests, but in two of the examples that you 
> mentioned, Sphinx and Jupyter, some manual inspection is necessary.

Now, at last, tell us what makes Sage so special that we must vendor
sphinx and jupyter (and pytest (proposed), and tox, and...), unlike,
say, sympy, or scipy?
I imagine they spend developers' time on something more productive
than repeating the work done elsewhere, no?

>
> A question to ask is what tooling is available to update the version pins, 
> and what the cost of using the tools is. For a typical upgrade, by improving 
> our tooling, we have reduced the work to just typing "./sage -package 
> update-latest sphinx --commit". In the Sphinx upgrade, 
> https://github.com/sagemath/sage/pull/37129/files (needs review), I ended up 
> updating 25 packages, so I had to use a command like this 25 times. It's 
> repetitive, maybe it takes 20 minutes total, but it's not remotely something 
> that I would use the phrase "Sage has shot itself in the foot" for.

The whole thing of a zillion vendored packages makes Sage uniquely
hard to package, and use outside of its own venv. These 25 packages
just don't need our version micromanagement, it's already done outside
of the project.
Can we please start to let go of this "vendor everything" mentality? Please?


>
> (Our tooling for "pip" packages is actually worse than that; "./sage -package 
> update-latest" does not support them, an easy to implement wishlist item. 
> Being able to run "sage -pip install -U sphinx", then test, then updating the 
> pinned versions according to "./sage -pip freeze" -- also that's an easy to 
> implement wishlist item.)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/7f062b45-a5b3-49de-83e1-4f2f47eb96c2n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq3AufJ0470DA1WYkCZd5eAcW7nwMC6eUhtLpjoMy91TDQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 11, 2024 at 3:34:46 PM UTC-8 Dima Pasechnik wrote:

On 11 February 2024 22:47:24 GMT, Matthias Koeppe  
wrote: 
>On Sunday, February 11, 2024 at 1:46:40 PM UTC-8 Matthias Koeppe wrote: 
> 
>I'll make an attempt to quantify this cost 
> 
>Here's an illustration of the workflow for making python_build a standard 
>"wheel" package, as proposed in 
>https://groups.google.com/g/sage-devel/c/MIU-xo9b7pc: 

What you outlined is the initial one-time cost. 

That's correct, that's what I did in that post.

There is also a cost of maintenance, which eventually gets bigger than the 
initial cost: the thing gets outdated, its dependencies get outdated, this 
all requires updates, tests, conflict resolutions ---something that you get 
largely for free if you let go of the package dependency micromanagement, 
relying instead on the Python universe out there to do the job.

That's where a possible sleight of hand happens. 
Let's please do this discussion at normal speed, giving the audience a 
chance to observe the facts and form their opinion.

Pinning packages to a set of tested working versions is a standard 
practice, and as a matter of fact part of best practices to achieve 
stability in various deployment situations, reproducibility, etc.

In the Python world, such pinning is done using requirements.txt, 
Pipfile.lock, and environment.yml files.
In the Sage distribution, we pin using package-version.txt and tiny 
requirements.txt files.

When updating the pins, testing is always necessary; it does not come for 
free. Yes, we have our automatic tests, but in two of the examples that you 
mentioned, Sphinx and Jupyter, some manual inspection is necessary.

A question to ask is what tooling is available to update the version pins, 
and what the cost of using the tools is. For a typical upgrade, by 
improving our tooling, we have reduced the work to just typing "./sage 
-package update-latest sphinx --commit". In the Sphinx upgrade, 
https://github.com/sagemath/sage/pull/37129/files (needs review), I ended 
up updating 25 packages, so I had to use a command like this 25 times. It's 
repetitive, maybe it takes 20 minutes total, but it's not remotely 
something that I would use the phrase "Sage has shot itself in the foot" 
for. 

(Our tooling for "pip" packages is actually worse than that; "./sage 
-package update-latest" does not support them, an easy to implement 
wishlist item. Being able to run "sage -pip install -U sphinx", then test, 
then updating the pinned versions according to "./sage -pip freeze" -- also 
that's an easy to implement wishlist item.)

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/7f062b45-a5b3-49de-83e1-4f2f47eb96c2n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-11 Thread Dima Pasechnik




On 11 February 2024 22:47:24 GMT, Matthias Koeppe  
wrote:
>On Sunday, February 11, 2024 at 1:46:40 PM UTC-8 Matthias Koeppe wrote:
>
>I'll make an attempt to quantify this cost
>
>
>Here's an illustration of the workflow for making python_build a standard 
>"wheel" package, as proposed in 
>https://groups.google.com/g/sage-devel/c/MIU-xo9b7pc:


What you outlined is the initial one-time cost. There is also a cost of 
maintenance, which eventually gets bigger than the initial cost: the thing gets 
outdated, its dependencies get outdated, this all requires updates, tests, 
conflict resolutions ---something that you get largely for free if you let go 
of the package dependency micromanagement, relying instead on the Python 
universe out there to do the job.




>
>
>*$ git checkout -b python_build_standard upstream/develop*branch 
>'python_build_standard' set up to track 'upstream/develop'.
>Switched to a new branch 'python_build_standard'
>
>*$ ls build/pkgs/python_build*SPKG.rst dependencies distros 
> requirements.txt type
>
>The package already exists as a "pip" package (requirements.txt). Let's 
>re-create it as a standard "wheel" package.
>
>
>
>*$ mv build/pkgs/python_build build/pkgs/build$ ./sage -package create 
>build --pypi --type standard*Downloading tarball from 
>https://pypi.io/packages/py3/b/build/build-1.0.3-py3-none-any.whl to 
>.../upstream/build-1.0.3-py3-none-any.whl
>[..]
>
>
>*$ mv build/pkgs/build build/pkgs/python_build$ ls 
>build/pkgs/python_build*SPKG.rst 
>checksums.inidependencies distros 
> install-requires.txt package-version.txt  requirements.txt type
>
>*$ git rm -f build/pkgs/python_build/requirements.txt*rm 
>'build/pkgs/python_build/requirements.txt'
>
>Now, after removing requirements.txt, it's a wheel package. Let's review 
>the changes that "sage -package create" made.
>
>
>*$ git --no-pager diff build/pkgs/python_build/dependencies*
>diff --git a/build/pkgs/python_build/dependencies 
>b/build/pkgs/python_build/dependencies
>index b72a6d1c776..47296a7bace 100644
>--- a/build/pkgs/python_build/dependencies
>+++ b/build/pkgs/python_build/dependencies
>@@ -1,4 +1,4 @@
>- pyparsing tomli packaging | $(PYTHON_TOOLCHAIN) $(PYTHON)
>+ | $(PYTHON_TOOLCHAIN) $(PYTHON)
> 
> --
> All lines of this file are ignored except the first.
>
>Our old version was better, go back to it. (The script "sage -package 
>create" does not know how to find the 
>dependencies; https://github.com/sagemath/sage/pull/36740 prepares an 
>improvement, needs review.)
>
>
>*$ git checkout -- build/pkgs/python_build/dependencies*
>
>Commit the changes:
>
>
>
>*$ git add build/pkgs/python_build$ git commit -m "build/pkgs/python_build: 
>Change to a normal standard package"*[python_build_standard 43f6b2b8ef9] 
>build/pkgs/python_build: Change to a normal standard package
> 4 files changed, 7 insertions(+), 1 deletion(-)
> create mode 100644 build/pkgs/python_build/checksums.ini
> rename build/pkgs/python_build/{requirements.txt => install-requires.txt} 
>(100%)
> create mode 100644 build/pkgs/python_build/package-version.txt
>
>Test it:
>
>
>*$ make python_build*make -j16 build/make/Makefile --stop
>./bootstrap -d
>[...]
>rm -rf config/install-sh config/compile config/config.guess 
>config/config.sub config/missing configure build/make/Makefile-auto.in
>make --no-print-directory python_build-SAGE_VENV-no-deps
>[python_build-1.0.3] Using cached file 
>.../upstream/build-1.0.3-py3-none-any.whl
>[python_build-1.0.3] python_build-1.0.3
>[python_build-1.0.3] 
>[python_build-1.0.3] Setting up build directory for python_build-1.0.3
>[...]
>[python_build-1.0.3] Using pip 23.3.1 from 
>.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/pip 
>(python 3.11)
>[python_build-1.0.3] Looking in links: 
>.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels
>[python_build-1.0.3] Processing 
>.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl
> 
>(from -r 
>.../local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/python_build/spkg-requirements.txt
> 
>(line 1))
>[python_build-1.0.3] Requirement already satisfied: packaging>=19.0 in 
>.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from 
>build@ 
>file://.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl->-r
> 
>.../local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/python_build/spkg-requirements.txt
> 
>(line 1)) (23.2)
>[python_build-1.0.3] Requirement already satisfied: pyproject_hooks in 
>.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from 
>build@ 
>file://.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl->-r
>

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-11 Thread Dima Pasechnik




On 11 February 2024 21:46:40 GMT, Matthias Koeppe  
wrote:
>I'll provide some context and pointers that readers may find helpful to 
>participate in the discussion and vote.
>- terminology: 
>https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-types
>- tooling: 
>https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages
>
>A "normal" or "wheel" package is always pinned to a specific version in the 
>Sage distribution (package-version.txt, checksums.ini), and the Sage 
>distribution needs to have a package for each of its dependencies.
>
>"Pip" packages can either be pinned to a specific version, or set 
>acceptable version ranges, or be entirely unconstrained. This is set in the 
>file requirements.txt in the package directory. 
>
>Pinning a version has the potential benefit of stability (avoiding 
>retroactive breakage by new, incompatible versions. The cost is that 
>updating the version requires work by two Sage developers: One who prepares 
>a PR and one who reviews it. (I'll make an attempt to quantify this cost in 
>a separate post.) And when the package does not get the attention of 
>developers who upgrade it, there's the potential risk of missing out on 
>bugfixes made in newer versions, or missing out on features in major new 
>versions.
>Not pinning the version has the obvious potential benefit of always being 
>up to date. But there is a risk of instability, either by the package 
>itself being affected by bugs in a new version, or by breaking 
>compatibility with Sage.

Sage had shot itself in the foot by adopting an overtly rigid approach to  
Python dependencies which are not  tightly integrated into the core of Sage 
(sagelib): Jupyter, Tox, and Sphinx (and their zillion dependencies).

A way out of it is to declare as many deps as possible pip, and just remove 
from our list many of  these packages which are dependencies of Sphinx and 
Jupyter only (they are found and installed by pip just fine when you install 
Jupyter and Sphinx, there is no need for Sage's micromanaging of them).
The potential issues with dependencies of pip packages interfering with Sage 
packages (you mention these below) are precisely the result of this package 
micromanagement.



>What policy is best for a package obviously depends on lots of factors, 
>including the development velocity and quality control that the upstream 
>project, interest by Sage developers in the package, the depth of 
>integration in Sage etc. I suggest to subject "one-size-fits-all" 
>approaches to a healthy dose of critical thinking.

Yes, indeed, the current "standard packages cannot be pip packages" is exactly 
"one-size-fits-all" approach you are arguing against, and the issue we would 
like to resolve here.

>
>Dependencies of a "pip" package do not need to be available as packages in 
>the Sage distribution. However, if a dependency is also a package of the 
>Sage distribution, then we must declare this dependency. If we don't, 
>surprising things can happen when building or upgrading. When new versions 
>of "pip" packages add dependencies that happen to be Sage packages, there 
>is a separate source of instability.

OTOH a package like pytest or tox is basically an external tool, and using an 
appropriate version of it is all what's needed.


>
>On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>
>> Currently the standard packages cannot be pip packages, i.e. we must, in 
>> effect, vendor them. This entails an extra effort which is often not 
>> needed, in particular as we patch only very few Python packages.
>> Pip packages are on the other hand installed straight from PyPI.
>>
>> Good examples of standard packages which can become pip ones are tox, 
>> pytest (not yet standard).
>>
>>
>> The other difference is that by default these packages are not included in 
>> the Sage releases source tarball.
>>
>> Rather than adding them there I propose to split the upstream/* part of 
>> the tarball into something optional - which is represented by a list of 
>> files to download, and which is just not needed if you build while 
>> connected to the internet.
>>
>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
>> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>>
>> Note that as William writes, the desire to have Sage buildable without an 
>> internet connection was a requirement by a past Sage funder, gone about 10 
>> years ago. Thus there's no longer an obligation to have this option.
>> I am not aware of a similar to Sage which provides tarballs allowing for 
>> an offline build.
>>
>> Thus, I would like to call a vote on these two topics:
>>
>> 1) allow standard packages to be pip packages 
>>
>> 2) drop the contents of upstream/ from the Sage source tarballs.
>>
>>
>> ---
>> Dima
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 11, 2024 at 1:46:40 PM UTC-8 Matthias Koeppe wrote:

I'll make an attempt to quantify this cost


Here's an illustration of the workflow for making python_build a standard 
"wheel" package, as proposed in 
https://groups.google.com/g/sage-devel/c/MIU-xo9b7pc:


*$ git checkout -b python_build_standard upstream/develop*branch 
'python_build_standard' set up to track 'upstream/develop'.
Switched to a new branch 'python_build_standard'

*$ ls build/pkgs/python_build*SPKG.rst dependencies distros 
 requirements.txt type

The package already exists as a "pip" package (requirements.txt). Let's 
re-create it as a standard "wheel" package.



*$ mv build/pkgs/python_build build/pkgs/build$ ./sage -package create 
build --pypi --type standard*Downloading tarball from 
https://pypi.io/packages/py3/b/build/build-1.0.3-py3-none-any.whl to 
.../upstream/build-1.0.3-py3-none-any.whl
[..]


*$ mv build/pkgs/build build/pkgs/python_build$ ls 
build/pkgs/python_build*SPKG.rst 
checksums.inidependencies distros 
 install-requires.txt package-version.txt  requirements.txt type

*$ git rm -f build/pkgs/python_build/requirements.txt*rm 
'build/pkgs/python_build/requirements.txt'

Now, after removing requirements.txt, it's a wheel package. Let's review 
the changes that "sage -package create" made.


*$ git --no-pager diff build/pkgs/python_build/dependencies*
diff --git a/build/pkgs/python_build/dependencies 
b/build/pkgs/python_build/dependencies
index b72a6d1c776..47296a7bace 100644
--- a/build/pkgs/python_build/dependencies
+++ b/build/pkgs/python_build/dependencies
@@ -1,4 +1,4 @@
- pyparsing tomli packaging | $(PYTHON_TOOLCHAIN) $(PYTHON)
+ | $(PYTHON_TOOLCHAIN) $(PYTHON)
 
 --
 All lines of this file are ignored except the first.

Our old version was better, go back to it. (The script "sage -package 
create" does not know how to find the 
dependencies; https://github.com/sagemath/sage/pull/36740 prepares an 
improvement, needs review.)


*$ git checkout -- build/pkgs/python_build/dependencies*

Commit the changes:



*$ git add build/pkgs/python_build$ git commit -m "build/pkgs/python_build: 
Change to a normal standard package"*[python_build_standard 43f6b2b8ef9] 
build/pkgs/python_build: Change to a normal standard package
 4 files changed, 7 insertions(+), 1 deletion(-)
 create mode 100644 build/pkgs/python_build/checksums.ini
 rename build/pkgs/python_build/{requirements.txt => install-requires.txt} 
(100%)
 create mode 100644 build/pkgs/python_build/package-version.txt

Test it:


*$ make python_build*make -j16 build/make/Makefile --stop
./bootstrap -d
[...]
rm -rf config/install-sh config/compile config/config.guess 
config/config.sub config/missing configure build/make/Makefile-auto.in
make --no-print-directory python_build-SAGE_VENV-no-deps
[python_build-1.0.3] Using cached file 
.../upstream/build-1.0.3-py3-none-any.whl
[python_build-1.0.3] python_build-1.0.3
[python_build-1.0.3] 
[python_build-1.0.3] Setting up build directory for python_build-1.0.3
[...]
[python_build-1.0.3] Using pip 23.3.1 from 
.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/pip 
(python 3.11)
[python_build-1.0.3] Looking in links: 
.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels
[python_build-1.0.3] Processing 
.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl
 
(from -r 
.../local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/python_build/spkg-requirements.txt
 
(line 1))
[python_build-1.0.3] Requirement already satisfied: packaging>=19.0 in 
.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from 
build@ 
file://.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl->-r
 
.../local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/python_build/spkg-requirements.txt
 
(line 1)) (23.2)
[python_build-1.0.3] Requirement already satisfied: pyproject_hooks in 
.../local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from 
build@ 
file://.../local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/build-1.0.3-py3-none-any.whl->-r
 
.../local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/python_build/spkg-requirements.txt
 
(line 1)) (1.0.0)
[python_build-1.0.3] Installing collected packages: build
[python_build-1.0.3]   changing mode of 
.../local/var/lib/sage/venv-python3.11/bin/pyproject-build to 755
[python_build-1.0.3] Successfully installed build-1.0.3
[python_build-1.0.3] Successfully installed python_build-1.0.3
[...]
Sage build/upgrade complete!

It did not complain about dependencies, so we seem to be good. But the 
"pyproject_hooks" that it was happy to find comes from the previous 
installation, we don't have it as a package. Let's create it as a standard 
package.


*$ ./sage -package

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

I'll provide some context and pointers that readers may find helpful to 
participate in the discussion and vote.
- terminology: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-types
- tooling: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages

A "normal" or "wheel" package is always pinned to a specific version in the 
Sage distribution (package-version.txt, checksums.ini), and the Sage 
distribution needs to have a package for each of its dependencies.

"Pip" packages can either be pinned to a specific version, or set 
acceptable version ranges, or be entirely unconstrained. This is set in the 
file requirements.txt in the package directory. 

Pinning a version has the potential benefit of stability (avoiding 
retroactive breakage by new, incompatible versions. The cost is that 
updating the version requires work by two Sage developers: One who prepares 
a PR and one who reviews it. (I'll make an attempt to quantify this cost in 
a separate post.) And when the package does not get the attention of 
developers who upgrade it, there's the potential risk of missing out on 
bugfixes made in newer versions, or missing out on features in major new 
versions.
Not pinning the version has the obvious potential benefit of always being 
up to date. But there is a risk of instability, either by the package 
itself being affected by bugs in a new version, or by breaking 
compatibility with Sage.
What policy is best for a package obviously depends on lots of factors, 
including the development velocity and quality control that the upstream 
project, interest by Sage developers in the package, the depth of 
integration in Sage etc. I suggest to subject "one-size-fits-all" 
approaches to a healthy dose of critical thinking.

Dependencies of a "pip" package do not need to be available as packages in 
the Sage distribution. However, if a dependency is also a package of the 
Sage distribution, then we must declare this dependency. If we don't, 
surprising things can happen when building or upgrading. When new versions 
of "pip" packages add dependencies that happen to be Sage packages, there 
is a separate source of instability.

On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:

> Currently the standard packages cannot be pip packages, i.e. we must, in 
> effect, vendor them. This entails an extra effort which is often not 
> needed, in particular as we patch only very few Python packages.
> Pip packages are on the other hand installed straight from PyPI.
>
> Good examples of standard packages which can become pip ones are tox, 
> pytest (not yet standard).
>
>
> The other difference is that by default these packages are not included in 
> the Sage releases source tarball.
>
> Rather than adding them there I propose to split the upstream/* part of 
> the tarball into something optional - which is represented by a list of 
> files to download, and which is just not needed if you build while 
> connected to the internet.
>
> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>
> Note that as William writes, the desire to have Sage buildable without an 
> internet connection was a requirement by a past Sage funder, gone about 10 
> years ago. Thus there's no longer an obligation to have this option.
> I am not aware of a similar to Sage which provides tarballs allowing for 
> an offline build.
>
> Thus, I would like to call a vote on these two topics:
>
> 1) allow standard packages to be pip packages 
>
> 2) drop the contents of upstream/ from the Sage source tarballs.
>
>
> ---
> Dima
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/b4df2e03-3eda-42a7-a7b7-07355b60c7d3n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-11 Thread mmarco

As I mentioned in the thread that motivated this one, it would be relevant 
to stablish if it is possible to move those packages from standard to pip, 
while still having a way to install sage without an internet connection.

If the effort is not too much, I think it would make sense to provide that 
alternative.
El domingo, 11 de febrero de 2024 a las 21:34:51 UTC+1, Matthias Koeppe 
escribió:

> On Sunday, February 11, 2024 at 12:26:41 PM UTC-8 Dima Pasechnik wrote:
>
>
> On 11 February 2024 19:50:17 GMT, Matthias Koeppe  
> wrote: 
> >I think it's a bit too quick to already call a vote. I would suggest that 
> >you take the time to collect and link previous discussions on this topic, 
> >so that participants can review the known arguments, viewpoints, and 
> >requirements. 
> > 
> >Example (from my previous 
> >post): 
> https://groups.google.com/g/sage-devel/c/C7-ho1zvEYU/m/S2n8d5rOAgAJ 
> >(2016) 
>
> I don't think arguments from 2016 are very relevant today, given how much 
> python packaging evolved since then.
>
>
> In case it was not clear, I did not suggest to only look for discussions 
> from 2016 or earlier.
>
> And the state of Python packaging is only one aspect that is relevant.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/84f991b2-8753-45e4-aab4-b7446bf7b357n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

On Sunday, February 11, 2024 at 12:26:41 PM UTC-8 Dima Pasechnik wrote:

On 11 February 2024 19:50:17 GMT, Matthias Koeppe  
wrote: 
>I think it's a bit too quick to already call a vote. I would suggest that 
>you take the time to collect and link previous discussions on this topic, 
>so that participants can review the known arguments, viewpoints, and 
>requirements. 
> 
>Example (from my previous 
>post): https://groups.google.com/g/sage-devel/c/C7-ho1zvEYU/m/S2n8d5rOAgAJ 
>(2016) 

I don't think arguments from 2016 are very relevant today, given how much 
python packaging evolved since then.

In case it was not clear, I did not suggest to only look for discussions 
from 2016 or earlier.

And the state of Python packaging is only one aspect that is relevant.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/962422f0-2d1b-416c-848b-a2535f9196c6n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

2024-02-11 Thread Dima Pasechnik




On 11 February 2024 19:50:17 GMT, Matthias Koeppe  
wrote:
>I think it's a bit too quick to already call a vote. I would suggest that 
>you take the time to collect and link previous discussions on this topic, 
>so that participants can review the known arguments, viewpoints, and 
>requirements.
>
>Example (from my previous 
>post): https://groups.google.com/g/sage-devel/c/C7-ho1zvEYU/m/S2n8d5rOAgAJ 
>(2016)


I don't think arguments from 2016 are very relevant today, given how much 
python packaging evolved since then.

I don't think there is a good reason to delay this vote, especially given that 
there is a pending vote on more
pip packages to be made standard, potentially leading to totally unneeded 
effort to vendor them.


>
>It may also be relevant to consider whether the "Source code (tar.gz)" 
>tarballs that are automatically provided by GitHub on releases (and tags) 
>would be sufficient. (They do not contain upstream; but they also do not 
>contain the helpful .git directory that our tarball release script 
>painstakingly adds.)
>
>On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote:
>
>> Currently the standard packages cannot be pip packages, i.e. we must, in 
>> effect, vendor them. This entails an extra effort which is often not 
>> needed, in particular as we patch only very few Python packages.
>> Pip packages are on the other hand installed straight from PyPI.
>>
>> Good examples of standard packages which can become pip ones are tox, 
>> pytest (not yet standard).
>>
>>
>> The other difference is that by default these packages are not included in 
>> the Sage releases source tarball.
>>
>> Rather than adding them there I propose to split the upstream/* part of 
>> the tarball into something optional - which is represented by a list of 
>> files to download, and which is just not needed if you build while 
>> connected to the internet.
>>
>> This is a huge saving on the tarball size: with upstream/* in, Sage 10.2 
>> tarball is 1.3Gb, and without it is smaller than 0.25Gb.
>>
>> Note that as William writes, the desire to have Sage buildable without an 
>> internet connection was a requirement by a past Sage funder, gone about 10 
>> years ago. Thus there's no longer an obligation to have this option.
>> I am not aware of a similar to Sage which provides tarballs allowing for 
>> an offline build.
>>
>> Thus, I would like to call a vote on these two topics:
>>
>> 1) allow standard packages to be pip packages 
>>
>> 2) drop the contents of upstream/ from the Sage source tarballs.
>>
>>
>> ---
>> Dima
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/C4BE24E6-3CAF-4E7E-ABB7-90BD95395963%40gmail.com.

[sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size