Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Matthias Koeppe]

On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe  
wrote:

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

  

2) The major improvement is that sagelib will be easier to install into an 
existing venv, and that's a wish of quite a number of users. Much more 
Pythonic, too.


The pip-installability of sagelib has absolutely nothing to do with this 
discussion.


Of course it does, a lot. As having less deps pinning will model 
installability and useability of sagelib in a "foreign" venv. At the moment 
using sagelib in a foreign venv is complicated and error-prone, and 
untested.


The dependencies of sagelib are declared in the template 
https://github.com/sagemath/sage/blob/develop/src/setup.cfg.m4#L13, which 
is filled with the data from the "install-requires.txt" files (such 
as 
https://github.com/sagemath/sage/blob/develop/build/pkgs/cysignals/install-requires.txt).
These files provide version ranges. They do NOT use the specific versions 
pinned in the Sage distribution.

I don't think it's necessary to comment much on the idea that we should 
change our production environment (unpinning the pinned versions of the 
Sage distribution) to make it "model installability and usability of 
sagelib" in a less controlled environment.

Testing sagelib (the sagemath-standard distribution package) in different 
environments is not "complicated" -- because I've built the convenient 
tooling for that. One does not have to make changes to the Sage 
distribution for that. 
See 
https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-standard/tox.ini#L18
 
for the different options. See also the developer's guide 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging_sage_library#testing-the-distribution-in-virtual-environments-with-tox,
 
where all of this is documented.


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/85a4614c-e003-461d-91f3-d43f498bfcf1n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On Sun, Feb 18, 2024 at 6:57 PM Nathan Dunfield 
wrote:

> On Sunday, February 18, 2024 at 12:14:58 PM UTC-6 Dima Pasechnik wrote:
>
> Reading:
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
>
>
> The wheel you talk about is just another packaging of a source package,
> isn't it?
>
>
> No.
>
>
> Well, I might have used incorrect terminology, but our wheels are always
> "build from source" wheels.
>
>
> No, many pure Python wheels are simply downloaded in the their final form
> from PyPI, for example:
>
> https://github.com/sagemath/sage/tree/develop/build/pkgs/webcolors
> https://github.com/sagemath/sage/tree/develop/build/pkgs/jupyterlab
>

OK, I am sorry, I've hallucinated a non-existing format for distributing
Python packages.
(or perhaps I saw somewhere a package which has both a pure-Python wheel,
and platform-specific wheels
with optional  binary extensions)


Wheels are platform-specific, and this non-pure-python packages need to
have platform-specific wheels,
see  e.g.
https://pypi.org/project/matplotlib/#files

>
>
> Best,
>
> Nathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/9775f89c-4901-4a0f-a0e0-91b38c215f18n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2_vfJJ97m55SV-wsULkGLvbzki%2Bw56Jxw%2BkGbGJ1UWTQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Nathan Dunfield]

On Sunday, February 18, 2024 at 12:14:58 PM UTC-6 Dima Pasechnik wrote:

Reading: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
 

The wheel you talk about is just another packaging of a source package, 
isn't it?


No.


Well, I might have used incorrect terminology, but our wheels are always 
"build from source" wheels.


No, many pure Python wheels are simply downloaded in the their final form 
from PyPI, for example:

https://github.com/sagemath/sage/tree/develop/build/pkgs/webcolors 
https://github.com/sagemath/sage/tree/develop/build/pkgs/jupyterlab

Best,

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9775f89c-4901-4a0f-a0e0-91b38c215f18n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On Sun, Feb 18, 2024 at 6:09 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:
>
> On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe 
> wrote:
>
> On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:
>
> 1) you can even just get a binary wheel of pytest installed - it is very
> fast, and robust.
>
>
> Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does.
> It installs pytest as a "wheel" package.
>
> there are wheels and wheels.
> Binary wheels don't need any building, Sage's wheel packages still do
> building from source - in case the package has C extensions, possibly
> cythonizing or running a similar built processs involving
> compilation/linking.
>
>
> No, Dima.
>
> Reading:
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
>
>
> The wheel you talk about is just another packaging of a source package,
> isn't it?
>
>
> No.
>

Well, I might have used incorrect terminology, but our wheels are always
"build from source" wheels.


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/e87b2425-e69c-42d6-8534-519e5eddaa63n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq00Et16AEfSdkNisi9tuGdps-cC2eTNvD-gxH-mXnovLQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On Sun, Feb 18, 2024 at 5:15 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 4:40:35 AM UTC-8 Dima Pasechnik wrote:
>
> On 17 February 2024 23:31:43 GMT, Matthias Koeppe 
> wrote:
> >On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote:
> >"wheel" type Sage packages, each of which is
> >primarily just the version number of a file on PyPI and its hash, is like
> a
> >"requirements.txt" file (or "conda-lock" file, for that matter) spread
> over
> >multiple directories. Personally, I don't view that as packaging a
> >dependency, but rather saving some metadata to aid
> >reliability/reproducibility.
> >
> >I'll note that in addition to aiding reliability/reproducibility, the
> >metadata in build/pkgs is also important for discoverability and
> >attribution.
>
> Merely pinning down the versions doesn't magically brings you
> reproducibility, unless you are also willing to pin down the OS version,
> and the hardware. [...]
>
>
> This is an instance of the "all or nothing" fallacy, and simultaneously a
> "straw man" fallacy (note that both Nathan and I said it "aids"
> reproducibility etc.)
>

I don't know where you see fallacies. You are just not willing to view the
existing Sage tooling under a critical loop,
IMHO.

>
> Besides, to create a pinning of all the versions of Python packages, just
> run the appropriate pip command, it will produce a full list of all the
> versions, ready to be used to reproduce the environment. No need to
> maintain these pinnings by hand.
>
> Yes, metadata is important, it's just make-work to maintain it manually.
> We don't need to carry out this make-work.
>
>
> Exactly, it matters what tooling is available. And every single little
> improvement of our tooling will likely have more value than this entire
> thread.
>

The value is in the eye of beholder, if you like.
The existing tooling is too rigid, my proposal aims to improve it by making
it more flexible.


>
> Yes, we don't want to maintain the metadata "manually".
> And we don't! We use the "sage --package" script (
> https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages
> )
>
> Yes, "pip freeze" will output the current versions of installed Python
> packages, to be saved as a requirements.txt file.
> What's missing is the tooling that would feed this version information
> back to our version files.
>

why do we need "our version files"? we need input to pip, and "pip freeze"
can generate such an input.



> That's wishlist item https://github.com/sagemath/sage/issues/37314,
> estimated effort: 15 minutes of work. Any takers?
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/16525787-4dec-4ded-b2c1-4019683a930an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0-tuzToFhvi7iNNxkgRTHYNSqSQrSewy%3DqfcHfVp%3DHwQ%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Matthias Koeppe]

On Sunday, February 18, 2024 at 10:05:21 AM UTC-8 Dima Pasechnik wrote:

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe  
wrote:

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

1) you can even just get a binary wheel of pytest installed - it is very 
fast, and robust.


Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does. It 
installs pytest as a "wheel" package. 

there are wheels and wheels.
Binary wheels don't need any building, Sage's wheel packages still do 
building from source - in case the package has C extensions, possibly 
cythonizing or running a similar built processs involving 
compilation/linking.


No, Dima.

Reading: 
https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#package-source-types
 

The wheel you talk about is just another packaging of a source package, 
isn't it?


No.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/e87b2425-e69c-42d6-8534-519e5eddaa63n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On Sun, Feb 18, 2024 at 5:24 PM Matthias Koeppe 
wrote:

> On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:
>
> 1) you can even just get a binary wheel of pytest installed - it is very
> fast, and robust.
>
>
> Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does.
> It installs pytest as a "wheel" package.
>
there are wheels and wheels.
Binary wheels don't need any building, Sage's wheel packages still do
building from source - in case the package has C extensions, possibly
cythonizing or running a similar built processs involving
compilation/linking.
The wheel you talk about is just another packaging of a source package,
isn't it?

Just the other day we saw how one can very well install pyscipopt, an
(optional) Sage package,
from a binary wheel, without building (cause Sage's Cython is too new for
pyscipopt, one can't in present build it from source).
https://groups.google.com/g/sage-devel/c/htVeuOH_3t8/m/mIjBH7aRAgAJ
Doing the same with e.g. scipy will shave off quite a bit  of build time
(by for this scipy needs to become a pip package).



> Whether you install a package from an sdist or from a wheel, you still
> have the same runtime dependencies ("install-requires").
> What goes away when you use a wheel is only the build-time dependencies
> ("build-system requires").
>

>
> 2) The major improvement is that sagelib will be easier to install into an
> existing venv, and that's a wish of quite a number of users. Much more
> Pythonic, too.
>
>
> The pip-installability of sagelib has absolutely nothing to do with this
> discussion.
>

Of course it does, a lot. As having less deps pinning will model
installability and useability of sagelib in a "foreign" venv. At the moment
using sagelib in a foreign venv is complicated and error-prone, and
untested.

And I should also say that less deps will make it easier for downstreams to
package sage/sagelib.



>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/3299fe86-fa67-4830-9f3c-1386235c37c8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0jkCCKzxhFGFLLsZYacUPzZU%2BhvkPUroKKc2XNBHTbeg%40mail.gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Matthias Koeppe]

On Sunday, February 18, 2024 at 9:07:04 AM UTC-8 Dima Pasechnik wrote:

1) you can even just get a binary wheel of pytest installed - it is very 
fast, and robust.


Yes, that's what my PR https://github.com/sagemath/sage/pull/37301 does. It 
installs pytest as a "wheel" package.

Whether you install a package from an sdist or from a wheel, you still have 
the same runtime dependencies ("install-requires"). 
What goes away when you use a wheel is only the build-time dependencies 
("build-system requires").
  

2) The major improvement is that sagelib will be easier to install into an 
existing venv, and that's a wish of quite a number of users. Much more 
Pythonic, too.


The pip-installability of sagelib has absolutely nothing to do with this 
discussion.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3299fe86-fa67-4830-9f3c-1386235c37c8n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Matthias Koeppe]

On Sunday, February 18, 2024 at 4:40:35 AM UTC-8 Dima Pasechnik wrote:

On 17 February 2024 23:31:43 GMT, Matthias Koeppe  
wrote: 
>On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote: 
>"wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like 
a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread 
over 
>multiple directories. Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility. 
> 
>I'll note that in addition to aiding reliability/reproducibility, the 
>metadata in build/pkgs is also important for discoverability and 
>attribution. 

Merely pinning down the versions doesn't magically brings you 
reproducibility, unless you are also willing to pin down the OS version, 
and the hardware. [...]


This is an instance of the "all or nothing" fallacy, and simultaneously a 
"straw man" fallacy (note that both Nathan and I said it "aids" 
reproducibility etc.)

Besides, to create a pinning of all the versions of Python packages, just 
run the appropriate pip command, it will produce a full list of all the 
versions, ready to be used to reproduce the environment. No need to 
maintain these pinnings by hand. 

Yes, metadata is important, it's just make-work to maintain it manually. We 
don't need to carry out this make-work.


Exactly, it matters what tooling is available. And every single little 
improvement of our tooling will likely have more value than this entire 
thread.

Yes, we don't want to maintain the metadata "manually". 
And we don't! We use the "sage --package" script 
(https://deploy-livedoc--sagemath.netlify.app/html/en/developer/packaging#utility-script-to-create-and-maintain-packages)

Yes, "pip freeze" will output the current versions of installed Python 
packages, to be saved as a requirements.txt file. 
What's missing is the tooling that would feed this version information back 
to our version files. That's wishlist 
item https://github.com/sagemath/sage/issues/37314, estimated effort: 15 
minutes of work. Any takers?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/16525787-4dec-4ded-b2c1-4019683a930an%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On 18 February 2024 15:51:27 GMT, Nathan Dunfield  wrote:
>On Sunday, February 18, 2024 at 6:26:28 AM UTC-6 Dima Pasechnik wrote:
>
>I cannot imagine CI breaking down by, say, pytest.
>
>
>I can definitely see that happening, and indeed it seems to have done so 
>for other projects:
>
>https://github.com/pytest-dev/pytest/issues/9765
>https://github.com/pytest-dev/pytest/issues/11983
> 

Well, yes, a major pytest version jump might be an issue. We know this well, so 
what?

>
>Besides, you can pin down or limit the version of a pip package, just as 
>well. E.g. pin down the version of Cython. But leave its dependencies out 
>of Sage, as much as possible.
>
>
>Leaving out dependencies doesn't eliminate complexity, it just hides it.  

Why would two dependencies of pytest* and pytest* only need to enter Sage 
codebase? No, there is no point in it, to the contrary it is more make-work, 
more resources, longer build times, etc.

>That doesn't seem like an improvement to me. 

well:

1) you can even just get a binary wheel of pytest installed - it is very fast, 
and robust. Just like you do with pytest being optional. There is no need for 
any extra deps getting in.

2) The major improvement is that sagelib will be easier to install into an 
existing venv, and that's a wish of quite a number of users. Much more 
Pythonic, too.


>  Plus, some packages are 
>dependencies of multiple other packages, and pip's ability to find a 
>version of said dependency that all other packages will tolerate is not the 
>greatest in my experience.

with Sage unnecessary pinning down a lot of package versions (and simply 
vendoring them,) you cannot expect pip to do too well. We just have to let go 
of many dependencies which are, needlessly, part of Sage, then pip will be able 
to do the job well.

Dima


>
>Best,
>
>Nathan
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9B2C761F-C865-4332-8106-6EA364A89FBD%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Nathan Dunfield]

On Sunday, February 18, 2024 at 6:26:28 AM UTC-6 Dima Pasechnik wrote:

I cannot imagine CI breaking down by, say, pytest.


I can definitely see that happening, and indeed it seems to have done so 
for other projects:

https://github.com/pytest-dev/pytest/issues/9765
https://github.com/pytest-dev/pytest/issues/11983
 

Besides, you can pin down or limit the version of a pip package, just as 
well. E.g. pin down the version of Cython. But leave its dependencies out 
of Sage, as much as possible.


Leaving out dependencies doesn't eliminate complexity, it just hides it.  
That doesn't seem like an improvement to me.  Plus, some packages are 
dependencies of multiple other packages, and pip's ability to find a 
version of said dependency that all other packages will tolerate is not the 
greatest in my experience.

Best,

Nathan



-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/73328e5d-6e5a-4245-823f-957a3b2ef230n%40googlegroups.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On 17 February 2024 23:31:43 GMT, Matthias Koeppe  
wrote:
>On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote:
>
>It seems to me that the "wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread over 
>multiple directories.  Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility.
>
>
>I agree with this viewpoint.
>I'll note that in addition to aiding reliability/reproducibility, the 
>metadata in build/pkgs is also important for discoverability and 
>attribution.

Merely pinning down the versions doesn't magically brings you reproducibility, 
unless you are also willing to pin down the OS version, and the hardware. It's 
often just doesn't make sence.
E.g. try to build Sage 8 or 9 on a modern machine with a recent OS. Chances are 
it won't work, despite you knowing all the versions.

Besides, to create a pinning of all the versions of Python packages, just run 
the appropriate pip command, it will produce a full list of all the versions, 
ready to be used to reproduce the environment. No need to maintain these 
pinnings by hand.

Yes, metadata is important, it's just make-work to maintain it manually. We 
don't need to carry out this make-work.


 
>
>Of course one could point out that our format is relatively verbose, as it 
>is spread over several plain-text files -- for simplicity of access with 
>shell scripts. Our format dates back to at least 2015, see the sage-devel 
>thread "Is there an online index of the standard packages shipped in Sage?" 
>(https://groups.google.com/g/sage-devel/c/aEmUmFOwJYQ/m/4pmmYrt3nXQJ) Other 
>distributions such as Pyodide 
>(https://github.com/pyodide/pyodide/tree/main/packages) use contemporary 
>structured datafile formats such as yaml instead. But it is probably safe 
>to say that most Sage users and even most Sage developers do not look at 
>the contents of the build/pkgs/ directory. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/14A14401-9869-4DA1-9C53-7C37A0FD5466%40gmail.com.

Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size

[Dima Pasechnik]

On 17 February 2024 23:04:49 GMT, Nathan Dunfield  wrote:
>On Saturday, February 17, 2024 at 1:13:33 PM UTC-6 Dima Pasechnik wrote:
>
>My proposal is in fact aimed at reducing the number of pinned Sage 
>dependecies, drastically. 
>
>Because most of them are either dependencies of Jupyterlab, or of Sphinx, 
>or of Python build system, and none of the them should be Sage's concern to 
>package, with all their dependencies.
>
>
>In my experience, it's particularly important to pin build dependencies.  
>Most of the "out of the blue" CI failures we've seen with "snappy" have 
>been caused by new versions of build dependencies, especially Cython. 


My proposal does not say that we must convert
all the standard python packages into pip packages.


I cannot imagine CI breaking down by, say, pytest.
Surely once every few years one might get a serious incompatibility, but not in 
10% of CI runs, or in even 1%.

Besides, you can pin down or limit the version of a pip package, just as well. 
E.g. pin down the version of Cython. But leave its dependencies out of Sage, as 
much as possible.




>  (I 
>see that Cython was also one of the motivations for the "conda-lock" scheme 
>of https://github.com/sagemath/sage/pull/35986 )
>
>If you itch to pack the said dependencies, please do it in a separate 
>repo/PyPI package, which can be consumed by sagelib to get the desired 
>pinned dependencies (and test all this in the existing CI, why not?) 
>But stop tying them up with sagelib - which in effect forces people 
>interested in sagelib to slave away on packaging 300 dependencies, most of 
>which aren't even tested by CI in any way, besides building.
>
>
>It seems to me that the "wheel" type Sage packages, each of which is 
>primarily just the version number of a file on PyPI and its hash, is like a 
>"requirements.txt" file (or "conda-lock" file, for that matter) spread over 
>multiple directories.  Personally, I don't view that as packaging a 
>dependency, but rather saving some metadata to aid 
>reliability/reproducibility.
>
>Best,
>
>Nathan
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/4F1F879D-76CB-4B59-A579-46868E4C8F23%40gmail.com.