Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On 27. 01. 22 15:54, Matti Picus wrote: On 27/1/22 14:59, Miro Hrončok wrote: ... The only remaining thing is libpypy3.9-c.so in /usr/bin. I'd like to move it to /usr/lib64 where it will be found. Technically, Fedora forbids an unversioned so file there. https://docs.fedoraproject.org/en-US/packaging-guidelines/#_downstream_so_name_versioning CPython has libpython3.9.so.1.0. $ objdump -p /usr/lib64/libpython3.9.so.1.0 | grep 'SONAME' SONAME libpython3.9.so.1.0 Do you think PyPy can do this as well? Or is there a reason not to? Thanks! (Previously replied privately, sorry about that). I tried moving the so to /usr/lib64/libpypy3-c.so.1.0 and adding a symlink to /usr/lib64/libpypy3-c.so. After running ldconfig to pick up the new file, I could run pypy3-c, and it seems venv and virtualenv do not complain about missing the shared object. So it seems it JustWorks. In order to be fully SONAME versioned, the file also needs to set SONAME in it. It seems it JustWorks as well: # mv /usr/bin/libpypy3.9-c.so /usr/lib64/libpypy3.9-c.so.1.0 # ln -s libpypy3.9-c.so.1.0 /usr/lib64/libpypy3.9-c.so # patchelf --set-soname libpypy3.9-c.so.1.0 /usr/lib64/libpypy3.9-c.so.1.0 # objdump -p /usr/lib64/libpypy3.9-c.so.1.0 | grep 'SONAME' SONAME libpypy3.9-c.so.1.0 # pypy3 Python 3.9.10 (5bee8a8dce85, Jan 27 2022, 11:29:45)... Awesome! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On 27/1/22 14:59, Miro Hrončok wrote: ... The only remaining thing is libpypy3.9-c.so in /usr/bin. I'd like to move it to /usr/lib64 where it will be found. Technically, Fedora forbids an unversioned so file there. https://docs.fedoraproject.org/en-US/packaging-guidelines/#_downstream_so_name_versioning CPython has libpython3.9.so.1.0. $ objdump -p /usr/lib64/libpython3.9.so.1.0 | grep 'SONAME' SONAME libpython3.9.so.1.0 Do you think PyPy can do this as well? Or is there a reason not to? Thanks! (Previously replied privately, sorry about that). I tried moving the so to /usr/lib64/libpypy3-c.so.1.0 and adding a symlink to /usr/lib64/libpypy3-c.so. After running ldconfig to pick up the new file, I could run pypy3-c, and it seems venv and virtualenv do not complain about missing the shared object. So it seems it JustWorks. Matti ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On 06. 12. 21 12:30, Miro Hrončok wrote: 2) Help make PyPy 3.9 more like CPython 3.9 in this regard. So, this is now: /usr ├── bin │ ├── libpypy3.9-c.so │ ├── pypy -> pypy3.9 │ ├── pypy3 -> pypy3.9 │ ├── pypy3.9 │ ├── python -> pypy3.9 │ ├── python3 -> pypy3.9 │ └── python3.9 -> pypy3.9 ├── include │ └── pypy3.9 │ └── *.h └── lib64 └── pypy3.9 ├── *.py ├── */ ├── *.pypy39-pp73-x86_64-linux-gnu.so ├── __pycache__ │ ├── *.pypy39.opt-1.pyc │ └── *.pypy39.pyc └── site-packages/ That looks reasonable good -- everything is namespaced or a symbolic link. The only remaining thing is libpypy3.9-c.so in /usr/bin. I'd like to move it to /usr/lib64 where it will be found. Technically, Fedora forbids an unversioned so file there. https://docs.fedoraproject.org/en-US/packaging-guidelines/#_downstream_so_name_versioning CPython has libpython3.9.so.1.0. $ objdump -p /usr/lib64/libpython3.9.so.1.0 | grep 'SONAME' SONAME libpython3.9.so.1.0 Do you think PyPy can do this as well? Or is there a reason not to? Thanks! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On 04. 12. 21 19:18, Matti Picus wrote: On 2/12/21 8:45 pm, Miro Hrončok wrote: Hello PyPy folks, I co-maintain the PyPy and CPython interpreter packages in Fedora Thanks for supporting PyPy We basically took the advice from https://doc.pypy.org/en/latest/build.html That documentation may now need some updating. Several different versions of pypy (e.g. 3.6 and 3.7) could be installed alongside each other, as long as only one of them "owned" /usr/bin/pypy3. This e.g. means that the standard library is installed in: /usr/lib64/pypy3.8/lib/pypy3.8/ Or the header files are in: /usr/lib64/pypy3.8/include/pypy3.8/ Double namespacing like this seems a bit overengineered, and since the idea was to resemble CPython, I was aiming at this setup: ... It seems reasonable that PyPy should follow CPython's lead in layout. Which would be the exact copy of our CPython setup | sed s/python/pypy/ Indeed, that is the goal. OK, let's focus on this goal for PyPy 3.9. However, I found 4 problems: 1) The PyPy install scheme only has "lib", not "lib64" I think we could sort this by a tiny patch that we used to carry in CPython 3.8 before configurable platlibdir was introduced in 3.9. If it is part of CPython 3.9, it should be part of PyPy 3.9 as well. Can you check that the latest nightly at https://buildbot.pypy.org/nightly/py3.9/ indeed no longer needs that change? The standard library is prepared for this, but when building CPython 3.9+, we pass --with-platlibdir=lib64 to configure. I wouldn't know where to pas this to PyPy build. 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version I don't know if we even need to ship this file. It seems deprecated. Maybe it can be in /usr/include/pypy3.8/PyPy.h? I was confused by this as well. As Armin pointed out, it was part of the older cffi embedding code, and I think it can be dropped. See https://foss.heptapod.net/pypy/pypy/-/issues/3608 Ack. 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3 I would expect it to be the other way around This makes sense. For testing, we need pypy3, but can create it as a symlink already as part of the build. I can easily change the symlink order downstream. But if done upstream, it would make more sense. 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced This is the one thing I cannot wrap my head around. I do not wish to install this file "globally" to /usr/bin This also makes sense and some ideas were floated in other responses to this thread. It would be best if someone with a plan could open an issue for discussion, and even better a PR. Well, I have a plan that would work for Fedora (have /usr/lib64/libpypy3.9.so(.1.0)), but I am not sure how to implement it. So, I am asking on this list: How should we package PyPy 3.8 for Fedora? Do we need to install it to /usr/lib64/pypy3.8/ prefix even thou most (but not all) of the directories within would contain a "pypy3.8" subdirectory? Or is there a better option I do not yet see? While it is too late for pypy3.8, now is the time to tweak pypy3.9. I am not sure what support you expect from PyPy to recreate the same layout as the one used in CPython. OK, so for me, there are now 2 problems :) 1) How to package PyPy 3.8. 2) Help make PyPy 3.9 more like CPython 3.9 in this regard. I suppose for (1) we can live with "weird" paths for a while. Are there patches we can upstream? Not yet. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On 2/12/21 8:45 pm, Miro Hrončok wrote: Hello PyPy folks, I co-maintain the PyPy and CPython interpreter packages in Fedora Thanks for supporting PyPy We basically took the advice from https://doc.pypy.org/en/latest/build.html That documentation may now need some updating. Several different versions of pypy (e.g. 3.6 and 3.7) could be installed alongside each other, as long as only one of them "owned" /usr/bin/pypy3. This e.g. means that the standard library is installed in: /usr/lib64/pypy3.8/lib/pypy3.8/ Or the header files are in: /usr/lib64/pypy3.8/include/pypy3.8/ Double namespacing like this seems a bit overengineered, and since the idea was to resemble CPython, I was aiming at this setup: ... It seems reasonable that PyPy should follow CPython's lead in layout. Which would be the exact copy of our CPython setup | sed s/python/pypy/ Indeed, that is the goal. However, I found 4 problems: 1) The PyPy install scheme only has "lib", not "lib64" I think we could sort this by a tiny patch that we used to carry in CPython 3.8 before configurable platlibdir was introduced in 3.9. If it is part of CPython 3.9, it should be part of PyPy 3.9 as well. Can you check that the latest nightly at https://buildbot.pypy.org/nightly/py3.9/ indeed no longer needs that change? 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version I don't know if we even need to ship this file. It seems deprecated. Maybe it can be in /usr/include/pypy3.8/PyPy.h? I was confused by this as well. As Armin pointed out, it was part of the older cffi embedding code, and I think it can be dropped. See https://foss.heptapod.net/pypy/pypy/-/issues/3608 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3 I would expect it to be the other way around This makes sense. For testing, we need pypy3, but can create it as a symlink already as part of the build. 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced This is the one thing I cannot wrap my head around. I do not wish to install this file "globally" to /usr/bin This also makes sense and some ideas were floated in other responses to this thread. It would be best if someone with a plan could open an issue for discussion, and even better a PR. So, I am asking on this list: How should we package PyPy 3.8 for Fedora? Do we need to install it to /usr/lib64/pypy3.8/ prefix even thou most (but not all) of the directories within would contain a "pypy3.8" subdirectory? Or is there a better option I do not yet see? While it is too late for pypy3.8, now is the time to tweak pypy3.9. I am not sure what support you expect from PyPy to recreate the same layout as the one used in CPython. Are there patches we can upstream? There are other repackagers: debian, BSD, and conda-forge, so any changes would need to be done with an eye toward not breaking their workflow as well. Matti ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On Thu, 2021-12-02 at 23:48 +0100, Armin Rigo wrote: > Hi, > > On Thu, 2 Dec 2021 at 21:10, Michał Górny wrote: > > > > 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced > > > > TBH I've never really understood the purpose of this file. > > We've stopped using it at some point and nothing really changed for us. > > It is needed for "embedding" situations, which nowadays means > https://cffi.readthedocs.io/en/latest/embedding.html. If we want to > go the full CPython way, we need to rename and move > "$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so" > and have /usr/bin/pypy3.8 be a program that is linked to > "libpypy38.so" instead of "$localdir/libpypy3-c.so". This would have > consequences for my own habits---e.g. the executable would no longer > find its .so if it lives simply in the same directory. Arguably not a > big issue :-) A freshly translated pypy would not work without > LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not. > The CPython solution to that annoyance is to compile statically by > default, if no option is passed to ./configure; you need to give the > --enable-shared option explicitly, and all distributions do. So if we > want to copy all of that for PyPy, we could. Well, I'm not sure how portable it is (but quite likely portable to all major platforms) but you can use something like -Wl,-rpath,$ORIGIN/../lib64 to make PyPy find the library in the correct directory relative to its executable. I think this shouldn't cause any major issues neither for running from extracted tarball, nor for system-wide installs. -- Best regards, Michał Górny ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On Thu, Dec 2, 2021 at 11:49 PM Armin Rigo wrote: > > https://cffi.readthedocs.io/en/latest/embedding.html. If we want to > go the full CPython way, we need to rename and move > "$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so" > and have /usr/bin/pypy3.8 be a program that is linked to > "libpypy38.so" instead of "$localdir/libpypy3-c.so". This would have > consequences for my own habits---e.g. the executable would no longer > find its .so if it lives simply in the same directory. Arguably not a > big issue :-) A freshly translated pypy would not work without > LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not. > I fear this would break much more than your own habits. To start with, it would break mine :) Jokes apart, it would break three big uses cases: 1. Download&run: if you want to try pypy, currently you can download the tarball, unpack and run, and it just works 2. non-privileged installation: currently you can install PyPy for your own user even if you don't have root power 3. install pypy in docker: this is basically the same as (1) but it's worth mentioning because a pattern which I saw a lot in real life production codie: it's very easy to install pypy in a Dockerfile, you just wget&unpack the tarball. If we require libpypy3.so so be in a system-wide directory, people will have to figure out themselves how to "install" pypy and/or we would need to provide an "install" script which will be very complex because of the zillions of slightly different details of linux distros So, I'm -1 to remove the possibility of $localdir/libpypy3-c.so. Maybe a solution which solves both problems is a compilation flag: 1. by default, we link to $localdir/libpypy3-c.so as we do now (and maybe we could even rename the file nowadats. The '-c' suffix is a relic of the past and it's probably no longer needed); 2. if you translate pypy with a special option, we link to a system-wide libpypy38.so, so that distros can compile a pypy which suits their needs Also, note that an poor's man (2) is already possible nowadays: we link to $localdir/libpypy3-c.so because pypy3-c it has an rpath set to $ORIGIN, but I think you can just patch the binary to remove it: https://stackoverflow.com/questions/13769141/can-i-change-rpath-in-an-already-compiled-binary ciao, Anto ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
Hi, On Thu, 2 Dec 2021 at 21:10, Michał Górny wrote: > > 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced > > TBH I've never really understood the purpose of this file. > We've stopped using it at some point and nothing really changed for us. It is needed for "embedding" situations, which nowadays means https://cffi.readthedocs.io/en/latest/embedding.html. If we want to go the full CPython way, we need to rename and move "$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so" and have /usr/bin/pypy3.8 be a program that is linked to "libpypy38.so" instead of "$localdir/libpypy3-c.so". This would have consequences for my own habits---e.g. the executable would no longer find its .so if it lives simply in the same directory. Arguably not a big issue :-) A freshly translated pypy would not work without LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not. The CPython solution to that annoyance is to compile statically by default, if no option is passed to ./configure; you need to give the --enable-shared option explicitly, and all distributions do. So if we want to copy all of that for PyPy, we could. (Note that the CPython explanation is based on my 2.7 knowledge and may be outdated.) A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
On Thu, 2021-12-02 at 19:45 +0100, Miro Hrončok wrote: > However, I found 4 problems: > > 1) The PyPy install scheme only has "lib", not "lib64" > > I think we could sort this by a tiny patch that we used to carry in CPython > 3.8 > before configurable platlibdir was introduced in 3.9. Are you trying to make PyPy multilib? Plain "lib" seems correct for libexecdir-style packages. > 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version > > I don't know if we even need to ship this file. It seems deprecated. Maybe it > can be in /usr/include/pypy3.8/PyPy.h? FWICS we're installing it there on Gentoo and we haven't had any problems because of it yet. > 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3 > > I would expect it to be the other way around, so both /usr/bin/pypy3.X and > /usr/bin/pypy3.Y can co-exist and /usr/bin/pypy3 is an (optional) symbolic > link > to one of them (as is the situtaion with CPython). I guess I can somehow > reverse the order manually, but I have not tested if it breaks anything. We're symlinking pypy3 -> pypy3-c-7.3.7 and it doesn't break anything, in case that helps. > 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced > > This is the one thing I cannot wrap my head around. I do not wish to install > this file "globally" to /usr/bin for two reasons: > > a) it conflicts between different PyPy Python versions > b) it ends up on $PATH but segfaults when executed TBH I've never really understood the purpose of this file. We've stopped using it at some point and nothing really changed for us. > So, I am asking on this list: How should we package PyPy 3.8 for Fedora? No, please don't, that would mean Gentoo would use its edge over Fedora ;-). -- Best regards, Michał Górny ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] Installation layout of the PyPy 3.8 Fedora package
Hello PyPy folks, I co-maintain the PyPy and CPython interpreter packages in Fedora and I am excited but a bit puzzled by a certain change that happened in PyPy 3.8. The installation layout of PyPy 3.8 was changed to be more like the CPython layout: https://doc.pypy.org/en/latest/release-v7.3.6.html > The 3.8 package now uses the same layout as CPython, and many of the > PyPy-specific changes to sysconfig, distutils.sysconfig, > and distutils.commands.install.py have been removed. The stdlib now is > located in /lib/pypy3.8 on posix systems, and in /Lib on Windows. > The include files on windows remain the same, on posix they are in > /include/pypy3.8 . Note we still use the pypy prefix to prevent mixing > the files with CPython (which uses python). Up until now (e.g. with PyPy 3.7) the installation layout of the Fedora package looked like this: /usr/lib64/pypy3.7/ ├── bin │ └── ... ├── include │ └── *.h ├── lib_pypy │ └── ... ├── lib-python │ └── 3 │ └── ... └── site-packages └── ... /usr/bin/pypy3.7 -> /usr/lib64/pypy3.7/bin/pypy3.7 (sometimes) /usr/bin/pypy3 -> /usr/bin/pypy3.7 We basically took the advice from https://doc.pypy.org/en/latest/build.html > To install PyPy system wide on unix-like systems, it is recommended to put > the whole hierarchy alone (e.g. in /opt/pypy) and put a symlink to the pypy > executable into /usr/bin or /usr/local/bin. Where we used "/usr/lib64/pypy3.7" instead of "/opt/pypy". Several different versions of pypy (e.g. 3.6 and 3.7) could be installed alongside each other, as long as only one of them "owned" /usr/bin/pypy3. Now with PyPy 3.8, if we are to keep this scheme, the entire tree gets a bit more nested: /usr/lib64/pypy3.8/ ├── bin │ └── ... ├── include │ ├── PyPy.h │ └── pypy3.8 │ └── *.h └── lib └── pypy3.8 ├── ... └── site-packages └── ... (/usr/bin setup remains the same) This e.g. means that the standard library is installed in: /usr/lib64/pypy3.8/lib/pypy3.8/ Or the header files are in: /usr/lib64/pypy3.8/include/pypy3.8/ Double namespacing like this seems a bit overengineered, and since the idea was to resemble CPython, I was aiming at this setup: /usr/ ├── bin │ └── ... ├── include │ └── pypy3.8 │ └── *.h └── lib(64) └── pypy3.8 ├── ... └── site-packages └── ... Which would be the exact copy of our CPython setup | sed s/python/pypy/ However, I found 4 problems: 1) The PyPy install scheme only has "lib", not "lib64" I think we could sort this by a tiny patch that we used to carry in CPython 3.8 before configurable platlibdir was introduced in 3.9. 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version I don't know if we even need to ship this file. It seems deprecated. Maybe it can be in /usr/include/pypy3.8/PyPy.h? 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3 I would expect it to be the other way around, so both /usr/bin/pypy3.X and /usr/bin/pypy3.Y can co-exist and /usr/bin/pypy3 is an (optional) symbolic link to one of them (as is the situtaion with CPython). I guess I can somehow reverse the order manually, but I have not tested if it breaks anything. 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced This is the one thing I cannot wrap my head around. I do not wish to install this file "globally" to /usr/bin for two reasons: a) it conflicts between different PyPy Python versions b) it ends up on $PATH but segfaults when executed So, I am asking on this list: How should we package PyPy 3.8 for Fedora? Do we need to install it to /usr/lib64/pypy3.8/ prefix even thou most (but not all) of the directories within would contain a "pypy3.8" subdirectory? Or is there a better option I do not yet see? Thanks for your help. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
