Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On Saturday, February 17, 2024 at 3:04:49 PM UTC-8 Nathan Dunfield wrote: It seems to me that the "wheel" type Sage packages, each of which is primarily just the version number of a file on PyPI and its hash, is like a "requirements.txt" file (or "conda-lock" file, for that matter) spread over multiple directories. Personally, I don't view that as packaging a dependency, but rather saving some metadata to aid reliability/reproducibility. I agree with this viewpoint. I'll note that in addition to aiding reliability/reproducibility, the metadata in build/pkgs is also important for discoverability and attribution. Of course one could point out that our format is relatively verbose, as it is spread over several plain-text files -- for simplicity of access with shell scripts. Our format dates back to at least 2015, see the sage-devel thread "Is there an online index of the standard packages shipped in Sage?" (https://groups.google.com/g/sage-devel/c/aEmUmFOwJYQ/m/4pmmYrt3nXQJ) Other distributions such as Pyodide (https://github.com/pyodide/pyodide/tree/main/packages) use contemporary structured datafile formats such as yaml instead. But it is probably safe to say that most Sage users and even most Sage developers do not look at the contents of the build/pkgs/ directory. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/df37948c-b153-4bbd-97a4-0a532b1778fen%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On Saturday, February 17, 2024 at 1:13:33 PM UTC-6 Dima Pasechnik wrote: My proposal is in fact aimed at reducing the number of pinned Sage dependecies, drastically. Because most of them are either dependencies of Jupyterlab, or of Sphinx, or of Python build system, and none of the them should be Sage's concern to package, with all their dependencies. In my experience, it's particularly important to pin build dependencies. Most of the "out of the blue" CI failures we've seen with "snappy" have been caused by new versions of build dependencies, especially Cython. (I see that Cython was also one of the motivations for the "conda-lock" scheme of https://github.com/sagemath/sage/pull/35986 ) If you itch to pack the said dependencies, please do it in a separate repo/PyPI package, which can be consumed by sagelib to get the desired pinned dependencies (and test all this in the existing CI, why not?) But stop tying them up with sagelib - which in effect forces people interested in sagelib to slave away on packaging 300 dependencies, most of which aren't even tested by CI in any way, besides building. It seems to me that the "wheel" type Sage packages, each of which is primarily just the version number of a file on PyPI and its hash, is like a "requirements.txt" file (or "conda-lock" file, for that matter) spread over multiple directories. Personally, I don't view that as packaging a dependency, but rather saving some metadata to aid reliability/reproducibility. Best, Nathan -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c1d9fe0f-c196-4fe3-b11f-2d8673f782a7n%40googlegroups.com.
[sage-devel] (Re-)building an inclusive SageMath community. I: How we talk about past, current, future contributions
Some brief thoughts that I would like to share. *A stated goal of our project is to be welcoming to new contributions and contributors.* "We invite newcomers to help improve the code or contribute in other ways. ... Contributions of all sorts are heartily welcomed ... The mission statement allows for a very wide variety of contributions." (https://github.com/sagemath/sage/blob/develop/CONTRIBUTING.md) To achieve these stated goals, not only do we need to generally ensure an atmosphere in the public forums of our community that is welcoming, but specifically *it matters how we talk about past contributions to the project*. Our CoC offers this insight: "Sage is constantly evolving, and earlier decisions that were made in good faith may sometimes need to be reconsidered. Nonetheless, we should still appreciate the hard work done in the past." (https://github.com/sagemath/sage/blob/develop/CODE_OF_CONDUCT.md) I feel that it is necessary to elaborate on this. Our project takes a long-term view on maintenance and contributions. It is, in fact, one of our project's more attractive propositions: "If you are willing to prepare your contribution so it meets our standards for inclusion [and making it ready is often a major investment...], then the project will be taking care of it in the long term, even after you have moved on to other things." For this proposition to work, contributors need to be able to trust that the project does not arbitrarily turn around just a couple a years later and declares the accepted contribution as dispensable, a waste of time, not really part of the project, etc. Our project's inclusive mission also means that *different people contribute to the project with a variety of motivations, constraints, expertise, and priorities.* It is important for everyone to remind themselves, e.g., that even if the time that they can set aside for Sage development may be very scarce, for others, e.g., hobbyists, it might simply not be a relevant constraint. Overall it's best to stay away from attempting to manage other developers' time and focus. (Also recall that in real life, it is supervisors who manage time and assignments of their subordinates...) *The community members who are perceived as part of the Sage project's leadership* have a particular responsibility in this regard. (This is not a "freedom of speech" issue; it is about responsible behavior. Words matter, community matters, leadership matters.) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/1279bc3f-d144-44b2-89aa-88f3c4c9a7e3n%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On Saturday, February 17, 2024 at 11:13:33 AM UTC-8 Dima Pasechnik wrote: On 17 February 2024 17:16:07 GMT, Matthias Koeppe wrote: >I share the same concern based on the amplification of the failure >probability, due to the large number of dependencies in Sage. My proposal is in fact aimed at reducing the number of pinned Sage dependecies, drastically. As you seem to be responding to what I wrote there, I have to point out that it is the *unpinning*, not the *pinning*, that is the concern for instability that Nathan and I share. Because most of them are either dependencies of Jupyterlab, or of Sphinx, or of Python build system, and none of the them should be Sage's concern to package, with all their dependencies. If you itch to pack the said dependencies, please do it in a separate repo/PyPI package, which can be consumed by sagelib to get the desired pinned dependencies (and test all this in the existing CI, why not?) But stop tying them up with sagelib - which in effect forces people interested in sagelib to slave away on packaging 300 dependencies, most of which aren't even tested by CI in any way, besides building. Please liberate sagelib from the cabal of the ftontend, etc. Sagemath is not a disto - no sane distro puts everything in one flat directory structure. Sagemath is an insane pile of needlessly vendored packages. Wow, there's so much to unpack here... -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/cf23d2b4-be5a-45f4-ac6d-f1922ce140f3n%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On 17 February 2024 17:16:07 GMT, Matthias Koeppe wrote: >On Saturday, February 17, 2024 at 7:06:27 AM UTC-8 Nathan Dunfield wrote: > >On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote: > >If one does not care about the use case without internet access, then it's >just the following: >- Pinning, as you mentioned (see also >https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above, >where I discussed some details of this, including risks of leaving packages >unpinned) >- Dependencies: "pip" packages can pull some of their build-time and >run-time dependencies directly from PyPI, without us mirroring these >dependencies in SageMath metadata. That's a mild convenience for >developers, of importance if one wants to leave the version range wide >open; but also has risks of instability. > > >Matthias, thanks for the clarification. I think pinning the version of a >"standard" package, including all its dependencies and down to the minor >release, is likely the best approach. Based on my experience with snappy >[1], not pinning things will result in CI runs failing "out of the blue" >because one of the dependencies got updated. With a small project like >snappy, this is pretty occasional and serves as a way to flag issues with >new upstream releases, but with Sage my guess is that such failures would >be frequent. Suppose that each time the CI runs on a new PR, there's a >10% chance of it failing because some completely unrelated dependency >shifted; that would be a major annoyance to seasoned Sage developers and >very discouraging to newcomers. > > >Thanks a lot, Nathan, for this point. >I share the same concern based on the amplification of the failure >probability, due to the large number of dependencies in Sage. My proposal is in fact aimed at reducing the number of pinned Sage dependecies, drastically. Because most of them are either dependencies of Jupyterlab, or of Sphinx, or of Python build system, and none of the them should be Sage's concern to package, with all their dependencies. If you itch to pack the said dependencies, please do it in a separate repo/PyPI package, which can be consumed by sagelib to get the desired pinned dependencies (and test all this in the existing CI, why not?) But stop tying them up with sagelib - which in effect forces people interested in sagelib to slave away on packaging 300 dependencies, most of which aren't even tested by CI in any way, besides building. Please liberate sagelib from the cabal of the ftontend, etc. Sagemath is not a disto - no sane distro puts everything in one flat directory structure. Sagemath is an insane pile of needlessly vendored packages. > >I'll note that recently we have merged PR >https://github.com/sagemath/sage/pull/35986 by Tobias Diez, which >implements full version pinning for our conda-forge installation methods. >See >https://github.com/sagemath/sage/blob/develop/src/environment-dev-3.11-macos-arm64.yml#L329 > >-- pytest is fully pinned down to the build hash. This change was motivated >specifically by the fast changes of conda-forge (as a rolling >distribution), which was one major cause for the instability of the CI >Conda workflow. >The tooling for this is https://conda.github.io/conda-lock/ and Tobias's >script >https://github.com/sagemath/sage/blob/develop/.github/workflows/conda-lock-update.py > >The proposed policy change to allow standard packages to be "pip" packages >would run in the opposite direction. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/F661E14B-700E-4DCF-959A-DED3EFFB5898%40gmail.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On 17 February 2024 15:01:14 GMT, Kwankyu Lee wrote: > > >there are ways to use pip without internet, with the necessary wheels >pre-fetched. >That's what Sage does with wheel packages. > > >Yes. This is a sage package of source type "wheel", as Matthias explained. > > >The difference between wheel packages vs pip packages is that the latter >don't require pre-fetched wheels, and absence of the need for package >(micro)management. > > >So your proposal is to let a standard' package to be installed by pip via >internet. Hence your proposal suggests to break the rule that >sage-the-distribution can be installed without internet connection. If so, >breaking the rule (that is, assuming internet connection at install time) >is more substantial content of your proposal. Once again: my proposal does not address the question how one can pack everything for an offline install. We suspect the demand for this feature does not exist (while the requirements for it are gone long ago), but there are options to create a fully offline installer readily available. Dima > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/3FDD96B0-313D-4334-A751-E5AA8AA402E8%40gmail.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On Saturday, February 17, 2024 at 7:06:27 AM UTC-8 Nathan Dunfield wrote: On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote: If one does not care about the use case without internet access, then it's just the following: - Pinning, as you mentioned (see also https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above, where I discussed some details of this, including risks of leaving packages unpinned) - Dependencies: "pip" packages can pull some of their build-time and run-time dependencies directly from PyPI, without us mirroring these dependencies in SageMath metadata. That's a mild convenience for developers, of importance if one wants to leave the version range wide open; but also has risks of instability. Matthias, thanks for the clarification. I think pinning the version of a "standard" package, including all its dependencies and down to the minor release, is likely the best approach. Based on my experience with snappy [1], not pinning things will result in CI runs failing "out of the blue" because one of the dependencies got updated. With a small project like snappy, this is pretty occasional and serves as a way to flag issues with new upstream releases, but with Sage my guess is that such failures would be frequent. Suppose that each time the CI runs on a new PR, there's a 10% chance of it failing because some completely unrelated dependency shifted; that would be a major annoyance to seasoned Sage developers and very discouraging to newcomers. Thanks a lot, Nathan, for this point. I share the same concern based on the amplification of the failure probability, due to the large number of dependencies in Sage. I'll note that recently we have merged PR https://github.com/sagemath/sage/pull/35986 by Tobias Diez, which implements full version pinning for our conda-forge installation methods. See https://github.com/sagemath/sage/blob/develop/src/environment-dev-3.11-macos-arm64.yml#L329 -- pytest is fully pinned down to the build hash. This change was motivated specifically by the fast changes of conda-forge (as a rolling distribution), which was one major cause for the instability of the CI Conda workflow. The tooling for this is https://conda.github.io/conda-lock/ and Tobias's script https://github.com/sagemath/sage/blob/develop/.github/workflows/conda-lock-update.py The proposed policy change to allow standard packages to be "pip" packages would run in the opposite direction. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c01e6162-99b2-44f0-9173-10ee199bd536n%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On Friday, February 16, 2024 at 11:17:37 PM UTC-6 Matthias Koeppe wrote: If one does not care about the use case without internet access, then it's just the following: - Pinning, as you mentioned (see also https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/9rF77fvFAAAJ above, where I discussed some details of this, including risks of leaving packages unpinned) - Dependencies: "pip" packages can pull some of their build-time and run-time dependencies directly from PyPI, without us mirroring these dependencies in SageMath metadata. That's a mild convenience for developers, of importance if one wants to leave the version range wide open; but also has risks of instability. Matthias, thanks for the clarification. I think pinning the version of a "standard" package, including all its dependencies and down to the minor release, is likely the best approach. Based on my experience with snappy [1], not pinning things will result in CI runs failing "out of the blue" because one of the dependencies got updated. With a small project like snappy, this is pretty occasional and serves as a way to flag issues with new upstream releases, but with Sage my guess is that such failures would be frequent. Suppose that each time the CI runs on a new PR, there's a 10% chance of it failing because some completely unrelated dependency shifted; that would be a major annoyance to seasoned Sage developers and very discouraging to newcomers. Now for a smaller package without (many?) dependencies, one can probably get away with pinning just down to the major release, but the benefit of doing that is pretty marginal. So I am against allowing standard packages to be "pip" packages --- the "wheel" approach seems like the right tradeoff here as (a) the final package is sourced straight off PyPI so we don't have to build it (b) the result is completely stable. Best, Nathan [1] https://snappy.computop.org -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5b0a8e66-3b2c-48df-8287-258d8252decdn%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
there are ways to use pip without internet, with the necessary wheels pre-fetched. That's what Sage does with wheel packages. Yes. This is a sage package of source type "wheel", as Matthias explained. The difference between wheel packages vs pip packages is that the latter don't require pre-fetched wheels, and absence of the need for package (micro)management. So your proposal is to let a standard' package to be installed by pip via internet. Hence your proposal suggests to break the rule that sage-the-distribution can be installed without internet connection. If so, breaking the rule (that is, assuming internet connection at install time) is more substantial content of your proposal. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6250e304-eb52-452b-9fc6-03121cdd57f7n%40googlegroups.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On 17 February 2024 02:26:32 GMT, Kwankyu Lee wrote: > > > > >By default the package content would be fetched, as pip does, > > >Not just as pip does, but by actually calling "pip" to contact PyPI. > > >and that would mean the default configuration for sage would require >internet at install time. > > >That's right. > > >Then Dima's proposal implies assuming internet at install time. Right? > >I asked the same question before. But Dima denied it. Whence I got >confused... there are ways to use pip without internet, with the necessary wheels pre-fetched. That's what Sage does with wheel packages. The difference between wheel packages vs pip packages is that the latter don't require pre-fetched wheels, and absence of the need for package (micro)management. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/E2A6AEBB-9885-48A9-BD83-AA5C913BACC4%40gmail.com.
Re: [sage-devel] Re: [Proposal] allow standard packages to be pip packages, reduce source tarball size
On 17 February 2024 02:26:32 GMT, Kwankyu Lee wrote: > > > > >By default the package content would be fetched, as pip does, > > >Not just as pip does, but by actually calling "pip" to contact PyPI. > > >and that would mean the default configuration for sage would require >internet at install time. > > >That's right. > > >Then Dima's proposal implies assuming internet at install time. Right? > >I asked the same question before. But Dima denied it. Whence I got >confused... > >-- >You received this message because you are subscribed to the Google Groups >"sage-devel" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to sage-devel+unsubscr...@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/sage-devel/63525bee-fdc8-4b8f-918e-ae475deae35en%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6CDD08D5-FB9F-4382-B8D0-9877A4C8A72F%40gmail.com.
Re: [sage-devel] Re: Proposal: Make pytest, pytest_xdist, pytest_mock, python_build standard packages
I have made it clear under what condition you can count my vote as +1. To make it clear: it is -1 if the condition is not met. On 17 February 2024 04:26:14 GMT, Matthias Koeppe wrote: >Note that posting a proposal here on sage-devel to make the packages >standard followed the policies of our project. >"optional packages stay in that status for at least a year, after which >they can be proposed to be included as standard packages in Sage. For this >a GitHub PR is opened with the label c: packages: standard. Then make a >proposal in the Google Group sage-devel." >https://doc.sagemath.org/html/en/developer/packaging.html#inclusion-procedure-for-new-and-updated-packages > >Multicasting "negative reviews" to tickets is not part of the procedures of >our project; >cf. https://github.com/sagemath/sage/pull/36726#issuecomment-1820148873 on >the emergence of this idea. > >On Friday, February 16, 2024 at 4:34:12 PM UTC-8 Dima Pasechnik wrote: > >> >> >> On 16 February 2024 23:33:48 GMT, Matthias Koeppe >> wrote: >> >On Friday, February 16, 2024 at 1:25:13 PM UTC-8 Dima Pasechnik wrote: >> > >> >>On Friday, February 16, 2024 at 9:38:16 AM UTC-8 Dima Pasechnik wrote: >> >My vote for is conditional on them remaining pip packages, and that's not >> >what your PRs do. >> > >> > >> >I'll count that as 0. >> >> you can count that as negative reviews on your PRs >> as they are now. >> >> >> > >> > >-- >You received this message because you are subscribed to the Google Groups >"sage-devel" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to sage-devel+unsubscr...@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/sage-devel/9151514d-395e-437c-ac65-b61dbfcd0ddbn%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/3B7884C5-A0E3-49D1-B0B8-0AE72A8AEF0C%40gmail.com.
