If we keep a "configure" tarball in each separate Sage installation but they share the rest of "upstream", then we save a lot of space and a lot of downloading. A workflow like
% make configure % configure --with-lots-of-options % make would be familiar and unchanged from the status quo when working with the git clone. Currently users don't have to run `./bootstrap` manually just to build Sage, and it would be nice to keep it this way. By the way, I just cloned the Sage repo and ran "make configure", which runs `./bootstrap`. The upstream directory is empty after that. If it's getting used for temporary storage for this tarball, that can be done elsewhere (.upstream.d? config? some temporary directory?). Another option for the location of a shared "upstream" would be Yet Another Environment Variable. On Monday, February 19, 2024 at 2:41:11 PM UTC-8 Dima Pasechnik wrote: > On Mon, Feb 19, 2024 at 10:29 PM Matthias Koeppe <matthia...@gmail.com> > wrote: > >> An option to "./configure" could work too, except that the "bootstrap" >> phase already downloads the "configure" tarball into that directory. > > > an option to ./bootstrap then would be logical > > >> >> Another possible direction: I've been thinking about creating a "./sage >> worktree" command, see https://github.com/sagemath/sage/issues/34744 >> >> >> On Monday, February 19, 2024 at 2:24:40 PM UTC-8 John H Palmieri wrote: >> >>> Regarding symlinking the upstream directory: instead or in addition, >>> what about an option to `./configure` for the location of that directory? >>> >>> >>> On Monday, February 19, 2024 at 12:09:54 PM UTC-8 Matthias Koeppe wrote: >>> >>>> Prompted by the discussion of space use on the *local machines* of >>>> users and developers, I propose another item in addition to A and B: >>>> >>>> *C. Advertise use of "git worktree" and recommend symlinking the >>>> "upstream" directory.* For testing a new release when you have an >>>> existing clone of the repository, using "git clone" another time is >>>> overkill as it creates another copy of the .git directory. And there is no >>>> point in having multiple copies of the "upstream" directory, as the >>>> filenames of the tarballs change whenever the contents change. >>>> >>>> On Monday, February 19, 2024 at 11:42:01 AM UTC-8 John H Palmieri wrote: >>>> >>>> This (A and B below) has the advantage of being quite explicit. The >>>> original proposal >>>> >>>> 1) allow standard packages to be pip packages >>>> >>>> 2) drop the contents of upstream/ from the Sage source tarballs. >>>> >>>> sounds explicit, but the more the discussion goes on, the more I feel >>>> that there are hidden pieces. Does the proposal also mean removing version >>>> restrictions and all of the other claimed maintenance burden for various >>>> components of Sage? >>>> >>>> Regarding item (2): if I clone the github repository, there is no >>>> upstream directory at the start, but after building Sage, it ends up being >>>> almost as large as in the current tarballs. (This is on OS X with a lot of >>>> homebrew packages installed.) So how much savings are we actually talking >>>> about? (Maybe it's not savings for the end user that are important, so >>>> *what* are we saving? Disk space on the mirrors?) Dima, can you please >>>> provide data? If we convert (according to (1)) to pip packages, those >>>> still >>>> need to be downloaded, and while they may not end up in "upstream" — I >>>> don't actually know how they work — don't they still take up disk space? >>>> So >>>> again, how much savings are we talking about? Please provide data. >>>> >>>> By the way, the git clone yields a package that is 616M on my machine. >>>> A good chunk of that is the .git directory. Are you proposing that we do >>>> not distribute this? (A recent beta tarball is 1.4G, unpacked 1.6G.) >>>> >>>> Regarding item (1): can you provide a list of packages that would >>>> become pip packages? Or describe how you would come up with a list? >>>> >>>> >>>> On Monday, February 12, 2024 at 3:52:29 PM UTC-8 Matthias Koeppe wrote: >>>> >>>> I'll now offer: >>>> >>>> *Opinion 1. Nobody needs to care in the slightest what the size of that >>>> release tarball is. * >>>> >>>> In any use cases with internet connectivity, people will be better off >>>> by just cloning the git repo, not use the release tarball. >>>> >>>> If there are relevant use cases without internet connectivity (I have >>>> no opinion to offer on this), then the release tarball has exactly the >>>> right contents. >>>> >>>> *Proposed action items: * >>>> *A.* Change https://github.com/sagemath/sage/blob/develop/README.md so >>>> that "git clone" is described as the primary way to obtain the Sage >>>> sources. That the big release tarball is available can be a footnote in >>>> the >>>> Installation Guide ( >>>> https://deploy-livedoc--sagemath.netlify.app/html/en/installation/source#installation-steps) >>>> >>>> for the limited no-internet connectivity use case. >>>> >>>> *B. *Likewise, get rid of all of these "Download Sage source code" >>>> pages (https://www.sagemath.org/download-source.html, >>>> https://www.sagemath.org/download-latest.html), mirror selection, etc. >>>> from the Sage website. >>>> >>>> >>>> On Sunday, February 11, 2024 at 11:23:42 AM UTC-8 Dima Pasechnik wrote: >>>> >>>> Currently the standard packages cannot be pip packages, i.e. we must, >>>> in effect, vendor them. This entails an extra effort which is often not >>>> needed, in particular as we patch only very few Python packages. >>>> Pip packages are on the other hand installed straight from PyPI. >>>> >>>> Good examples of standard packages which can become pip ones are tox, >>>> pytest (not yet standard). >>>> >>>> >>>> The other difference is that by default these packages are not included >>>> in the Sage releases source tarball. >>>> >>>> Rather than adding them there I propose to split the upstream/* part of >>>> the tarball into something optional - which is represented by a list of >>>> files to download, and which is just not needed if you build while >>>> connected to the internet. >>>> >>>> This is a huge saving on the tarball size: with upstream/* in, Sage >>>> 10.2 tarball is 1.3Gb, and without it is smaller than 0.25Gb. >>>> >>>> Note that as William writes, the desire to have Sage buildable without >>>> an internet connection was a requirement by a past Sage funder, gone about >>>> 10 years ago. Thus there's no longer an obligation to have this option. >>>> I am not aware of a similar to Sage which provides tarballs allowing >>>> for an offline build. >>>> >>>> Thus, I would like to call a vote on these two topics: >>>> >>>> 1) allow standard packages to be pip packages >>>> >>>> 2) drop the contents of upstream/ from the Sage source tarballs. >>>> >>>> >>>> --- >>>> Dima >>>> >>>> -- >> > You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-devel+...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/sage-devel/ab2b37cb-e44c-4489-bb88-8ba834064661n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/sage-devel/ab2b37cb-e44c-4489-bb88-8ba834064661n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/4182ebb4-d5f1-4a73-a6b9-18054eab900bn%40googlegroups.com.
