Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Kwankyu Lee]

Please Dima, let me stop this quoting business from your private messages.

Dima's response:

> It is perfectly avoidable, as I explained in more details, which you have 
chosen not to copy here.

OK. Then which one of these quotes from you is the details explaining how 
to install sage from source without internet connection by "pip" packages?
 
(1) > You can specify a collection of pip packages to download by pip (cf. 
"pip download") for installing along with the Sage tarball, if you must.

(2) > ... actually, with pip-installable Sage, you can simply tell pip to 
install 
from a github repo. Conditional on all the libraries already installed, 
that's how it all can work for development purposes - and for installation 
of a stable sagemath, one can just use PyPI.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3fda9128-b4b9-4d7f-9657-ba1a2174c9acn%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Kwankyu Lee]

OK. Sage tarballs and the wheels are uploaded to our mirror sites. So wheel 
packages end up in the mirror sites. This is what you say "mirroring PyPI".

You object to "mirroring PyPI", that is, wheel packages. For its 
replacement, you propose to fetch wheels directly from PyPI by switching to 
"pip" packages.
For users installing sage from sage tarball, this (switching to "pip" 
packages) means (again) to assume internet connection at install time. 
Right?


This is Dima's reponse to this:

> You still need to get the tarball from the internet, no? 

Once a user gets the tarball either from the internet or by usb stick, he 
can install sage from the tarball later without internet connection, say in 
flight.

If we keep this installation form as the standard way to install sage from 
source and do not give up tarball, then I think wheel packages and 
"mirroring PyPI" is unavoidable.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/cc37e358-75c0-4391-8013-55f2c4acac6en%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Kwankyu Lee]

I explained to Kwankyu that I mean "mirroring" in the usual sense of this 
word, not the one he thought.


OK. Sage tarballs and the wheels are uploaded to our mirror sites. So wheel 
packages end up in the mirror sites. This is what you say "mirroring PyPI".

You object to "mirroring PyPI", that is, wheel packages. For its 
replacement, you propose to fetch wheels directly from PyPI by switching to 
"pip" packages.
For users installing sage from sage tarball, this (switching to "pip" 
packages) means (again) to assume internet connection at install time. 
Right?

After some thought, now 

(A) "assuming internet connection for users installing from sage tarball" 

sounds a sort of oxymoron, because (A) defeats the purpose of providing 
tarball in the first place. Why tarball? It is for users with no internet 
connection. Otherwise, you can simply download sage source from github by 
git clone and the wheels from PyPI.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/46b5c78c-ba99-4cf7-9119-723ee7bad1f7n%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Matthias Koeppe]

On Monday, June 10, 2024 at 11:10:34 AM UTC-7 Dima Pasechnik wrote:

We are discussing your (general) proposal to provide binary wheels for 
packages by mirroring PyPI. 
That is where gigabytes come from. 
E.g. the binary wheels for scipy (with one version of Python) add up to 
about 200Mb.


Nobody proposed to make a change to how we install scipy.

I will not continue to respond in this thread.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/2883dd95-76ee-440d-be1e-2fc1f9c17ef5n%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On 10 June 2024 17:27:55 BST, Matthias Koeppe  wrote:
>On Monday, June 10, 2024 at 7:52:09 AM UTC-7 Kwankyu Lee wrote:
>
>This is Dima's response:
>> [...] multi-Gigabyte  territory.
>
>
>For a fact-based discussion, people may want to look at the actual size of 
>the wheels of the only rust-based package that we are discussing. 
>https://pypi.org/project/rpds-py/#files (look for files *-cp*-.whl)
>It's just a few megabytes total.
rpds-py is rather small.

We are discussing your (general) proposal to provide binary wheels for packages 
by mirroring PyPI.
That is where gigabytes come from.
E.g. the binary wheels for scipy (with one version of Python) add up to about 
200Mb.

Dima
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9FB76EAA-8368-4B08-9CC9-335134AE7F56%40gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On 10 June 2024 17:21:16 BST, Matthias Koeppe  wrote:
>On Monday, June 10, 2024 at 7:37:48 AM UTC-7 Kwankyu Lee wrote:
>
>So, why do we need to mirror PyPI ?  
>
>
>I understand "mirroring PyPI"  as what we do with "wheel" packages, that 
>is, delivering the wheel (downloaded from PyPI) of the specified version in 
>the sage tarball.
>
>
>Exactly.

Exactly, what?

 I explained to Kwankyu that I mean "mirroring" in the usual sense of this 
word, not the one he thought.

Dima


>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/DC52175F-7A77-490E-915A-33163B52BE42%40gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On Mon, Jun 10, 2024 at 3:52 PM Kwankyu Lee  wrote:
>
> This is Dima's response:
>
> > None are relevant:
> > (1) can be trivially achieved without mirroring.
> > (2) is irrelevant here, as creating a tarball with all these binary
> > wheels pushes us into multi-Gigabyte  territory.
> > That is, you'd need prefetch the set of binary wheels for your machine
> > somehow - and again, for this you don't need to mirror PyPI.
>
> In my own words, he seems to be saying that "we don't need to download the 
> wheel and put it into our tarball; we can just call pip to fetch the 
> specified version from internet at install time". To do what he is 
> advocating, all we have to do is to throw away the "no internet for 
> installing from tarball" assumption.
>
> Did I understand you correctly, Dima?

Yes and no: namely, using the proposed mirrored binary wheels will throw away
"no internet for installing from tarball" assumption. just as well.
So there is no difference at this point.

Dima
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/f5ff66fd-6164-496b-9b4b-39663ce23eb5n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq3M-cMYyPOOQ3qaf3cDOvxA1rA7xcsDY6q21--0sFyHPw%40mail.gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On Mon, Jun 10, 2024 at 2:18 PM Kwankyu Lee  wrote:
>
> To add  variation to this boring discussion, let me intervene:
>
>
> So, why do we need to mirror PyPI ?
>
>
> From this discussion, I guess that the answer is
>
> 1. We want to pin the version of standard package
> 2. We do not want to assume internet connection
>
> Is there other reason, Matthias?
>
> Dima, which of (1) and (2) do you think we can abolish?
>
None are relevant:

(1) can be trivially achieved without mirroring.

(2) is irrelevant here, as creating a tarball with all these binary
wheels pushes us into multi-Gigabyte  territory.
That is, you'd need prefetch the set of binary wheels for your machine
somehow - and again, for this you don't need to mirror PyPI.

Dima

>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/b47a3e26-31e6-4755-8a7d-20e6e471aa9en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq32ymR_Kz6zp3wEzRkrszg%2B8Yr6u7i%3DaAsOFTsVaFKfZA%40mail.gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On 4 June 2024 18:46:44 BST, kcrisman  wrote:
>
>
>Yes. I am all for removing "no internet connection" assumption in 
>installing the sage distribution from source.
>
>
>Since we don't ship binaries, I'd like to hear from those who have had to 
>install Sage in circumstances with slow/no internet situations about this 
>(e.g. several of our French developers have had to do so in Francophone 
>Africa, if I recall correctly).  Perhaps this is no longer an issue, but it 
>would be nice to have confirmation.  I do agree that the issue of "certain 
>customers" needing no-internet installation should no longer be a 
>consideration.

There is absolutely no need to create huge tarfiles if you have to work offline 
with a collection of Pyhton packages (e.g. the collection of Sage packages); 
there are tools out there to creates offline mirrors of a collection of PyPI 
packages, such as Morgan:


Dima

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CF683C3C-5A4B-4E75-A579-196B795A4F53%40gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[kcrisman]

Yes. I am all for removing "no internet connection" assumption in 
installing the sage distribution from source.


Since we don't ship binaries, I'd like to hear from those who have had to 
install Sage in circumstances with slow/no internet situations about this 
(e.g. several of our French developers have had to do so in Francophone 
Africa, if I recall correctly).  Perhaps this is no longer an issue, but it 
would be nice to have confirmation.  I do agree that the issue of "certain 
customers" needing no-internet installation should no longer be a 
consideration.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/96770e9a-58ff-470f-8397-631f255ffb28n%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Kwankyu Lee]

There is no need to remove it - it suffices to convert it to a pip package. 
(yes, for this we need to allow standard pip packages - as I have been 
proposing).


Yes. I am all for removing "no internet connection" assumption in 
installing the sage distribution from source.
 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/a3064565-3310-497d-88c3-ec9c417b0bc5n%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On Monday, June 3, 2024 at 9:10:56 PM UTC+1 Matthias Koeppe wrote:

On Monday, June 3, 2024 at 12:41:41 PM UTC-7 Michael Orlitzky wrote:

Rust is not nearly as portable as C, and has an unstable ABI that makes 
shipping compatible versions of packages from multiple sources nearly 
impossible.


I share this concern about ABI compatibility, and would therefore for now 
like a policy that restricts the use of such binary packages to
- optional packages 
- semi-standard, i.e., "disablable", packages such as the Jupyter frontend.

How about not inventing more "semi-", "quasi-", etc. here, and just go on 
with my proposal to allow standard pip packages?

Dima 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/1551dead-28ac-4041-9a48-b6f1764940bfn%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Dima Pasechnik]

On 3 June 2024 23:47:14 CEST, Kwankyu Lee  wrote:
>
>
>Likewise, we will soon add support for installing Python packages from 
>platform-dependent wheels. This is needed for updating some Jupyter 
>components that have started to use Rust (https://github.com/crate-py/rpds, 
>a dependency of jsonschema).
>
>
>First sorry for an off-topic comment. If we need a feature (installing 
>Python packages from platform-dependent wheels) just to maintain jupyter, 
>then I think we should consider first removing jupyter from sage packages. 
>As far as I know, there is no technical reason why we have to ship jupyter, 
>we do it just because sage aims for "batteries included" assuming no 
>internet connection. But jupyter is too big to carry as a battery. 

There is no need to remove it - it suffices to convert it to a pip package. 
(yes, for this we need to allow standard pip packages - as I have been 
proposing).

Removing won't be very easy, as some parts of Jupyter are used by Sphinx. 
(doable, yes).

Dima

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/48839DF8-E517-4754-8D4E-6ABEC8C12A04%40gmail.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Michael Orlitzky]

On Mon, 2024-06-03 at 12:54 -0700, Matthias Koeppe wrote:
> 
> Could you share details regarding this? I'm not sure who "we" is in what 
> you write, but in the last Jupyter PR that I prepared, I had to use some 
> older versions of some packages to avoid pulling in the Rust dependency at 
> this point.

The one I have in mind is: I am peacefully using borgbackup, a python
program, to back up our servers.

Borg depends on pkgconfig (python). Pkgconfig uses poetry (python) as
its build system. But poetry has a dependency on jsonschema (python),
and one day, the jsonschema package adds a dependency on rpds (rust).
Now I need to rebuild rust packages every time I update my system in
order to use my python backup program. This requires a rust compiler,
ABI unstable, so not only do I have to update the packages that have
changed but pretty much every rust package on the system during every
upgrade.

Poetry is a popular build system so this impacted many casual users of
python programs. It was a major PITA on Gentoo in particular because
python runs on more architectures than rust. Lots of end user packages
"broke" when the rust dependency was introduced because on certain
arches those dependencies "disappeared."

Anyway, poetry eventually replaced jsonschema with fastjsonschema to
avoid cursing its own users:

  https://github.com/python-poetry/poetry-core/pull/642

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/7f7b2ddd34b4144b6f51a1762093e4e3e45e1bf8.camel%40orlitzky.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Matthias Koeppe]

On Monday, June 3, 2024 at 12:41:41 PM UTC-7 Michael Orlitzky wrote:

Rust is not nearly as portable as C, and has an unstable ABI that makes 
shipping compatible versions of packages from multiple sources nearly 
impossible.


I share this concern about ABI compatibility, and would therefore for now 
like a policy that restricts the use of such binary packages to
- optional packages 
- semi-standard, i.e., "disablable", packages such as the Jupyter frontend.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/d39341cf-3523-4e45-a71c-f25804f188e2n%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Matthias Koeppe]

On Monday, June 3, 2024 at 12:41:41 PM UTC-7 Michael Orlitzky wrote:

On Sat, 2024-06-01 at 10:02 -0700, Matthias Koeppe wrote: 
> we will soon add support for installing Python packages from 
> platform-dependent wheels. This is needed for updating some Jupyter 
> components that have started to use Rust (https://github.com/crate-py/rpds, 

> a dependency of jsonschema). 

The jsonschema package was being replaced by fastjsonschema in its 
consumers the last time we ran in to this problem.


Could you share details regarding this? I'm not sure who "we" is in what 
you write, but in the last Jupyter PR that I prepared, I had to use some 
older versions of some packages to avoid pulling in the Rust dependency at 
this point.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/020c7d6b-0725-4ac5-87ad-3fa2ce03902dn%40googlegroups.com.

Re: [sage-devel] Re: Current status of possibility of integrating libraries written in Rust into Sage

[Michael Orlitzky]

On Sat, 2024-06-01 at 10:02 -0700, Matthias Koeppe wrote:
> Unlikely that we would add a package to the Sage distribution that builds a 
> Rust library from source. 
> 
> Not so long ago we added support for installing Python packages from 
> platform-independent wheels. We did this to sidestep the concern of 
> shipping more and more of Javascript (Node.js) infrastructure that is 
> needed for building JupyterLab components.
> 
> Likewise, we will soon add support for installing Python packages from 
> platform-dependent wheels. This is needed for updating some Jupyter 
> components that have started to use Rust (https://github.com/crate-py/rpds, 
> a dependency of jsonschema).

The jsonschema package was being replaced by fastjsonschema in its
consumers the last time we ran in to this problem.

Rust is not nearly as portable as C, and has an unstable ABI that makes
shipping compatible versions of packages from multiple sources nearly
impossible. Will Sage's wheel packages be compatible with the wheels
built by the Gentoo package manager? Even for a single source, more
labor is involved -- instead of rebuilding each package once when it
changes, you wind up rebuilding all of its consumers, and then all of
their consumers, and... so on recursively.

We're already drowning in this regard and incorporating rust in any way
will make it a lot worse.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/4cbea3e620e0481d4d2849fec72b6e4c2d4e5dcc.camel%40orlitzky.com.