Neuro SIG Python packaging strategy change
Hi all, We, at the Neuro SIG, have been discussing a change in strategy related to packaging Python software. We maintain ~500 packages at the moment, about 300 are Python. It's a lot of work, and we're struggling to keep up. There's still a lot of software out there we'd like to "support" on Fedora too. https://pagure.io/neuro-sig/NeuroFedora/issue/580 Our primary target audience is researchers/scientists. The goal is for them to be able to use software on Fedora easily. What we're no longer sure about is whether providing system packages is the best way to achieve this, given that anecdotal evidence suggests that most people will set up virtual environments etc. and install packages directly from pypi. I am not aware of any users at all that rely on system packages (including me nowadays). So, we're thinking of: - prioritising packaging/maintaining software that is unavailable on PyPi (or difficult to install from there) - dropping software that is pip installable from Fedora - shifting focus to testing that the software we care about is pip installable on current Fedora releases - documenting software status (pip installable or not) and any additional tips/steps required to get it working and informing upstream of issues that we encounter and submitting fixes where possible This is the general plan. Folks are, of course, welcome to maintain packages that they think are useful as system packages, for example tools that provide command line interfaces and so on. If possible, perhaps we can run simple import tests to check that pip installed packages also "function correctly". I don't know how much work it would be to install stuff using pip and figuring out a way to get its code (from the source repo) and running unit tests etc.---but that's something we thought we can look into later if we have the resources. For the simple "check whether X installs on Fedora via pip", I thought perhaps "tmt" could be used, but I haven't looked into it in detail yet. https://docs.fedoraproject.org/en-US/ci/tmt/ What do you all think about this? Please share any thoughts you have about this, and any tips/tricks/tools that can be used. If it's something we can do for Python software in Fedora in general, we'd be happy to do that too. -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature -- ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Neuro SIG Python packaging strategy change
On Wed, Jun 25, 2025 14:37:49 +0200, Michael J Gruber wrote: > Ankur Sinha via python-devel venit, vidit, dixit 2025-06-25 14:19:28: > ... >> > > Thanks for this proposal. I think it's coming just at the right time and > should - as you indicate - help us focus on Fedora as a platform. In > particular: > > - The separation between "platform" and "application" wil become more > and more important when the platform is provided as an immutable OS > (for more users). Neither layering nor rolling your own (bootc) image > make much sense for an "app", nor (in many cases) a flatpak. > > - Reducing our set of rpm-packaged python apps helps whenever there is a > major py change, be it a mass rebuild for a new python version or a > change in packaging (such as dismissal of `setup.py install`). > > I like the idea of having a "Fedora vetted/tested subset of PyPi", i.e. > packages confirmed to be installable on Fedora X for all X in current > releases, a bit like koschei for PyPi, possibly using existing `%check` > or such. More ecosystems than just Python could benefit from this. Yes, it should be extensible to other ecosystems too. We'll try to hack something up as a prototype and report back on how it goes. -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature -- ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Neuro SIG Python packaging strategy change
Hi all, Quick update. I tinkered with two different methods of: - checking if a package installs on Fedora with pip - checking if it can be imported - checking if the provided command line tools work One was using tmt, the other using pytest. - tmt: https://pagure.io/neuro-sig/NeuroFedora/pull-request/582 - pytest: https://pagure.io/neuro-sig/NeuroFedora/pull-request/581 I think the pytest system works better because: - I couldn't figure out if tmt can do things in parallel - tmt doesn't seem to have the same configurability in terms of parameterized tests and so on (or again, I haven't been able to figure out how to do it properly) For the pytest workflow, one simply needs to add a json file for the package with some information: - name of the pypi package - extra packages - imports to check - clis to check The tests will pick up this info and run the tests. We've started to orphan packages and add them to the pytest testing framework. You can see this in the PR. So far we've not run into a package that is not installable/usable via pip on py3.12. I need to run checks with other supported python versions. I'm trying to figure out if that's something that can be passed as a parameter to the pytests too: > $ pytest -n auto -v > > test session starts > = > platform linux -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0 -- > /home/asinha/.local/share/virtualenvs/neurofedora-check/bin/python > cachedir: .pytest_cache > rootdir: > /home/asinha/Documents/02_Code/00_mine/NeuroFedora/pagure/python-package-usage-check > plugins: xdist-3.8.0, shutil-1.8.1, virtualenv-1.8.1 > 16 workers [14 items] > scheduling tests via LoadScheduling > > test_package.py::TestFedoraPythonPackage::test_installing[packages/mne] > test_package.py::TestFedoraPythonPackage::test_installing[packages/neat] > test_package.py::TestFedoraPythonPackage::test_installing[packages/netpyne] > test_package.py::TestFedoraPythonPackage::test_installing[packages/neurotune] > test_package.py::TestFedoraPythonPackage::test_installing[packages/pybv] > test_package.py::TestFedoraPythonPackage::test_installing[packages/nilearn] > test_package.py::TestFedoraPythonPackage::test_installing[packages/mne-bids] > test_package.py::TestFedoraPythonPackage::test_installing[packages/bluepyopt] > test_package.py::TestFedoraPythonPackage::test_installing[packages/pynn] > test_package.py::TestFedoraPythonPackage::test_installing[packages/pyneuroml] > test_package.py::TestFedoraPythonPackage::test_installing[packages/neuron] > test_package.py::TestFedoraPythonPackage::test_installing[packages/nipype] > test_package.py::TestFedoraPythonPackage::test_installing[packages/libneuroml] > test_package.py::TestFedoraPythonPackage::test_installing[packages/lfpy] > [gw11] [ 7%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/pybv] > [gw7] [ 14%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/neuron] > [gw13] [ 21%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/pynn] > [gw2] [ 28%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/libneuroml] > [gw8] [ 35%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/neurotune] > [gw4] [ 42%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/mne] > [gw1] [ 50%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/lfpy] > [gw9] [ 57%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/nilearn] > [gw6] [ 64%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/netpyne] > [gw0] [ 71%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/bluepyopt] > [gw3] [ 78%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/mne-bids] > [gw5] [ 85%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/neat] > [gw10] [ 92%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/nipype] > [gw12] [100%] PASSED > test_package.py::TestFedoraPythonPackage::test_installing[packages/pyneuroml] > > > 14 passed in 46.60s > = I'm running these locally in a virtual env for the moment. This works fine. I don't know if there's the possibility of running this on Fedora infra somewhere. We'll look into that later. -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature -- ___ python-devel mailing list -- python-deve
Re: Neuro SIG Python packaging strategy change
Hi Fabio, On Tue, Jul 08, 2025 16:48:34 +0200, Fabio Valentini via python-devel wrote: > On Tue, Jul 8, 2025 at 12:51 PM Ankur Sinha via python-devel > wrote: > > > > We've started to orphan packages and add them to the pytest testing > > framework. You can see this in the PR. So far we've not run into a > > package that is not installable/usable via pip on py3.12. I need to run > > checks with other supported python versions. I'm trying to figure out if > > that's something that can be passed as a parameter to the pytests too: > > I use `tox` for this purpose, it works great (we use it for the > rust2rpm / cargo2rpm upstream tests). > > For example, here's the configuration for rust2rpm: > https://pagure.io/fedora-rust/rust2rpm/blob/main/f/tox.ini#_1-16 > > With that configuration, running `tox p` runs the default tests for > the defined environments in parallel (in this case, running `pytest` > in parallel in venvs with py311, py312, py313, py314 interpreters, > respectively). Thanks for that. Sandro also pointed out that we should use tox on our matrix channel. I've updated our branch to do this now: https://pagure.io/neuro-sig/NeuroFedora/pull-request/581#request_diff -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature -- ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Neuro SIG Python packaging strategy change
Hi all, Quick update: We've begun to drop packages and test their installation/imports instead. The test are being run using tox/pytest: https://pagure.io/neuro-sig/NeuroFedora/blob/main/f/python-package-usage-check I'm running this every night for the moment, and results are being uploaded here: https://ankursinha.fedorapeople.org/neurofedora/package-status/ We're updating our docs here with a list of packages that we test. For example: https://docs.fedoraproject.org/en-US/neurofedora/compneuro-tools/ This seems to be working well enough for us at the moment :) -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature -- ___ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
