Re: Building the python executable statically with libpython*.a for better performance
- Original Message - > From: "Neal Gompa" > To: "Fedora Python SIG" > Cc: "David Gray" > Sent: Saturday, October 26, 2019 5:01:52 PM > Subject: Re: Building the python executable statically with libpython*.a for > better performance > > On Sat, Sep 21, 2019 at 5:16 PM Miro Hrončok wrote: > > > > Hello, > > > > we've been recently approached by a colleague from Red Hat working on > > performance (CCed). > > > > According to their testing, Fedora Python performance could be improved by > > ~15% > > by building /usr/bin/python* statically with libpython*.a. That sounds like > > a > > worthy thing to do. > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1749479 > > > > Since Python 3.8 Python extension modules are no longer linked to > > libpython.so > > and we can do the following: > > > > * build /usr/bin/python3(.8) statically with libpython*.a > > * build and ship libpython3.8.so.1.0 for packages that "embed" Python > > > > The change in the python3 package is trivial: > > > > https://src.fedoraproject.org/rpms/python3/pull-request/133 > > > > However it can have serious impact on Python extension modules that are > > linked > > to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by > > compiling > > code for Python extension module and code that embeds Python into one file. > > > > We will likely propose a Fedora 32 Change for this, however I'm opening > > this > > topic for discussion before we do so. > > > > Testing the proposed Pull Request with your code is also helpful. Let me > > know > > how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 > > python38 > > package with this change). > > > > WDYT? > > One major concern: We currently rely on the libpython shared object > linkage for non-standard builds to ensure we get everything for moving > to new Python versions. I've personally experienced quirks with trying > to force linking via the shared object when the Python interpreter > itself is not. For example, perl-Inline-Python determines how to build > itself by asking the interpreter. In Fedora, this works correctly > because the interpreter returns the shared object. In other > distributions where I've built the module, it doesn't and uses the > static link object, which makes it difficult to automatically generate > the link dependency to the system Python we are using. > > In Mageia, we actually reverted Python's change to not link libpython > with extension modules so that we still have the dependency link for > binary objects: > https://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/link-C-modules-with-libpython.patch?view=markup > > Is there no other way to get better performance? Do we want to > potentially give up the easily trackable dependency web for that? Is > it worth breaking non-standard build mechanisms that interrogate the > interpreter for how to link to it? > > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > python-devel mailing list -- python-devel@lists.fedoraproject.org > To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org > Hello. For the record most of the packages build fine with the change, including perl-Inline-Python [0]. Also still waiting for the change to be sent by the wrangler, but you can read the finalized version here: https://fedoraproject.org/wiki/Changes/PythonStaticSpeedup [0] https://copr.fedorainfracloud.org/coprs/g/python/Python3_statically_linked/builds/ -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Re: Building the python executable statically with libpython*.a for better performance
On 2019-10-26 17:01, Neal Gompa wrote: On Sat, Sep 21, 2019 at 5:16 PM Miro Hrončok wrote: Hello, we've been recently approached by a colleague from Red Hat working on performance (CCed). According to their testing, Fedora Python performance could be improved by ~15% by building /usr/bin/python* statically with libpython*.a. That sounds like a worthy thing to do. https://bugzilla.redhat.com/show_bug.cgi?id=1749479 Since Python 3.8 Python extension modules are no longer linked to libpython.so and we can do the following: * build /usr/bin/python3(.8) statically with libpython*.a * build and ship libpython3.8.so.1.0 for packages that "embed" Python The change in the python3 package is trivial: https://src.fedoraproject.org/rpms/python3/pull-request/133 However it can have serious impact on Python extension modules that are linked to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by compiling code for Python extension module and code that embeds Python into one file. We will likely propose a Fedora 32 Change for this, however I'm opening this topic for discussion before we do so. Testing the proposed Pull Request with your code is also helpful. Let me know how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 python38 package with this change). WDYT? One major concern: We currently rely on the libpython shared object linkage for non-standard builds to ensure we get everything for moving to new Python versions. I've personally experienced quirks with trying to force linking via the shared object when the Python interpreter itself is not. For example, perl-Inline-Python determines how to build itself by asking the interpreter. In Fedora, this works correctly because the interpreter returns the shared object. In other distributions where I've built the module, it doesn't and uses the static link object, which makes it difficult to automatically generate the link dependency to the system Python we are using. In Mageia, we actually reverted Python's change to not link libpython with extension modules so that we still have the dependency link for binary objects: https://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/link-C-modules-with-libpython.patch?view=markup Is there no other way to get better performance? Do we want to potentially give up the easily trackable dependency web for that? Is it worth breaking non-standard build mechanisms that interrogate the interpreter for how to link to it? I'll share a more high-level view, since I don't know too many details about linking. Hope y'all won't mind such vague text. Is it worth breaking non-standard build mechanisms that interrogate the interpreter for how to link to it? There's a spectrum: on one side, there's forever supporting any hack that happened to work the way some frustrated engineer put it in a cmake file. On the other, there's a mythical future-proof "How to link to Python" guide explaining the various use cases and the best ways to do things, which everyone follows. Both sides of the spectrum lead to stagnation. The middle would have some best practices but also room to experiment. I think interrogating the interpreter for how to link to it is an important use case. But it currently seems like it's a bit of black magic: Python exposes some of its internals in a bunch of different ways, and that's what everyone uses. Nobody knows how to do it properly (including authors of python-config). Things mostly work, but when the internals need to change (for a 20% speedup), some stuff will fail. I think Fedora is actually a great place to start changing that, because this is an integration problem, and distros are all about integration. Fedora has a lot of software with all kinds of needs and assumptions, we have pretty strong ties to upstreams (certainly CPython, and hopefully the other projects), so we can hopefully make changes that aren't one-sided, but coordinated across the ecosystem (assuming that the Fedora package collection is a reasonable sample of the ecosystem). So let's, figure out what people need and what's the best way to solve their problems. And if we can't answer all the questions, hit the revert button. Specifically: - What should code that both embeds and extends Python do? Do we have a concrete use case for this? - Since perl-Inline-Python was mentioned, could we explicitly make sure it works, or figure out what modifications it will need? ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Re: Building the python executable statically with libpython*.a for better performance
On 30. 10. 19 1:27, Neal Gompa wrote:
On Tue, Oct 29, 2019 at 6:58 PM Charalampos Stratakis
wrote:
Through our latest benchmarks the speedup could be up to 27% so I would say
it's definitely worth it, and we plan to work with the affected packages to see
how to best resolve the issues, if they arise, on a case by case basis.
Currently on rawhide there are 113 packages requiring libpython.
Ubuntu and Debian are doing this for years so I would expect that at least the
packages that are also included there, would either be working or require
slight modifications.
The change will be posted soon for F32 on the devel list, so you can check all
the details and we can continue this conversation by then.
Debian/Ubuntu packaging of Python works quite a bit differently. They
do install-time byte-compilation, and they made modifications
throughout the stack to make it as hard as possible to create a
dependency on a specific minor version of Python. We do exactly the
opposite (for VERY good reasons), and it's important to keep that in
mind.
Python performance improvement by nearly 30% is no joke, but we need
to figure out how to account for non-standard builds before
implementing it.
We do. Hower I'm nto sure I understand this specific problem that started this
thread. Can we workaround it by manually adding:
Requires: python(abi) = %{python3_version}
To the affected packages?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Re: Building the python executable statically with libpython*.a for better performance
On Tue, Oct 29, 2019 at 6:58 PM Charalampos Stratakis wrote: > > Through our latest benchmarks the speedup could be up to 27% so I would say > it's definitely worth it, and we plan to work with the affected packages to > see how to best resolve the issues, if they arise, on a case by case basis. > Currently on rawhide there are 113 packages requiring libpython. > > Ubuntu and Debian are doing this for years so I would expect that at least > the packages that are also included there, would either be working or require > slight modifications. > > The change will be posted soon for F32 on the devel list, so you can check > all the details and we can continue this conversation by then. > Debian/Ubuntu packaging of Python works quite a bit differently. They do install-time byte-compilation, and they made modifications throughout the stack to make it as hard as possible to create a dependency on a specific minor version of Python. We do exactly the opposite (for VERY good reasons), and it's important to keep that in mind. Python performance improvement by nearly 30% is no joke, but we need to figure out how to account for non-standard builds before implementing it. -- 真実はいつも一つ！/ Always, there's only one truth! ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Re: Building the python executable statically with libpython*.a for better performance
- Original Message - > From: "Neal Gompa" > To: "Fedora Python SIG" > Cc: "David Gray" > Sent: Saturday, October 26, 2019 5:01:52 PM > Subject: Re: Building the python executable statically with libpython*.a for > better performance > > On Sat, Sep 21, 2019 at 5:16 PM Miro Hrončok wrote: > > > > Hello, > > > > we've been recently approached by a colleague from Red Hat working on > > performance (CCed). > > > > According to their testing, Fedora Python performance could be improved by > > ~15% > > by building /usr/bin/python* statically with libpython*.a. That sounds like > > a > > worthy thing to do. > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1749479 > > > > Since Python 3.8 Python extension modules are no longer linked to > > libpython.so > > and we can do the following: > > > > * build /usr/bin/python3(.8) statically with libpython*.a > > * build and ship libpython3.8.so.1.0 for packages that "embed" Python > > > > The change in the python3 package is trivial: > > > > https://src.fedoraproject.org/rpms/python3/pull-request/133 > > > > However it can have serious impact on Python extension modules that are > > linked > > to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by > > compiling > > code for Python extension module and code that embeds Python into one file. > > > > We will likely propose a Fedora 32 Change for this, however I'm opening > > this > > topic for discussion before we do so. > > > > Testing the proposed Pull Request with your code is also helpful. Let me > > know > > how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 > > python38 > > package with this change). > > > > WDYT? > > One major concern: We currently rely on the libpython shared object > linkage for non-standard builds to ensure we get everything for moving > to new Python versions. I've personally experienced quirks with trying > to force linking via the shared object when the Python interpreter > itself is not. For example, perl-Inline-Python determines how to build > itself by asking the interpreter. In Fedora, this works correctly > because the interpreter returns the shared object. In other > distributions where I've built the module, it doesn't and uses the > static link object, which makes it difficult to automatically generate > the link dependency to the system Python we are using. > > In Mageia, we actually reverted Python's change to not link libpython > with extension modules so that we still have the dependency link for > binary objects: > https://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/link-C-modules-with-libpython.patch?view=markup > > Is there no other way to get better performance? Do we want to > potentially give up the easily trackable dependency web for that? Is > it worth breaking non-standard build mechanisms that interrogate the > interpreter for how to link to it? > Hi Neal and thanks for voicing that concern, Through our latest benchmarks the speedup could be up to 27% so I would say it's definitely worth it, and we plan to work with the affected packages to see how to best resolve the issues, if they arise, on a case by case basis. Currently on rawhide there are 113 packages requiring libpython. Ubuntu and Debian are doing this for years so I would expect that at least the packages that are also included there, would either be working or require slight modifications. The change will be posted soon for F32 on the devel list, so you can check all the details and we can continue this conversation by then. > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > python-devel mailing list -- python-devel@lists.fedoraproject.org > To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org > -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Re: Building the python executable statically with libpython*.a for better performance
On Sat, Sep 21, 2019 at 5:16 PM Miro Hrončok wrote: > > Hello, > > we've been recently approached by a colleague from Red Hat working on > performance (CCed). > > According to their testing, Fedora Python performance could be improved by > ~15% > by building /usr/bin/python* statically with libpython*.a. That sounds like a > worthy thing to do. > > https://bugzilla.redhat.com/show_bug.cgi?id=1749479 > > Since Python 3.8 Python extension modules are no longer linked to libpython.so > and we can do the following: > > * build /usr/bin/python3(.8) statically with libpython*.a > * build and ship libpython3.8.so.1.0 for packages that "embed" Python > > The change in the python3 package is trivial: > > https://src.fedoraproject.org/rpms/python3/pull-request/133 > > However it can have serious impact on Python extension modules that are linked > to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by > compiling > code for Python extension module and code that embeds Python into one file. > > We will likely propose a Fedora 32 Change for this, however I'm opening this > topic for discussion before we do so. > > Testing the proposed Pull Request with your code is also helpful. Let me know > how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 > python38 > package with this change). > > WDYT? One major concern: We currently rely on the libpython shared object linkage for non-standard builds to ensure we get everything for moving to new Python versions. I've personally experienced quirks with trying to force linking via the shared object when the Python interpreter itself is not. For example, perl-Inline-Python determines how to build itself by asking the interpreter. In Fedora, this works correctly because the interpreter returns the shared object. In other distributions where I've built the module, it doesn't and uses the static link object, which makes it difficult to automatically generate the link dependency to the system Python we are using. In Mageia, we actually reverted Python's change to not link libpython with extension modules so that we still have the dependency link for binary objects: https://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/link-C-modules-with-libpython.patch?view=markup Is there no other way to get better performance? Do we want to potentially give up the easily trackable dependency web for that? Is it worth breaking non-standard build mechanisms that interrogate the interpreter for how to link to it? -- 真実はいつも一つ！/ Always, there's only one truth! ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Building the python executable statically with libpython*.a for better performance
Hello, we've been recently approached by a colleague from Red Hat working on performance (CCed). According to their testing, Fedora Python performance could be improved by ~15% by building /usr/bin/python* statically with libpython*.a. That sounds like a worthy thing to do. https://bugzilla.redhat.com/show_bug.cgi?id=1749479 Since Python 3.8 Python extension modules are no longer linked to libpython.so and we can do the following: * build /usr/bin/python3(.8) statically with libpython*.a * build and ship libpython3.8.so.1.0 for packages that "embed" Python The change in the python3 package is trivial: https://src.fedoraproject.org/rpms/python3/pull-request/133 However it can have serious impact on Python extension modules that are linked to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by compiling code for Python extension module and code that embeds Python into one file. We will likely propose a Fedora 32 Change for this, however I'm opening this topic for discussion before we do so. Testing the proposed Pull Request with your code is also helpful. Let me know how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 python38 package with this change). WDYT? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
