Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
On Sunday, 01 December 2019 at 00:07, Zbigniew Jędrzejewski-Szmek wrote: > On Sat, Nov 30, 2019 at 05:26:14PM +0100, Dominik 'Rathann' Mierzejewski > wrote: [...] > > Two observations: > > > > 1. It actually generates more BRs than I specify manually (I had 3). It > > adds python3-wheel and it brings in python3-pip and python3-pytoml on it > > own, so my package ends up with 4 additional BRs for no apparent gain. > > BR or R? If it just a few additional BR on small python packages, I don't > it matters much. BR and it matters in this case, because it increases the number of BRs by 100%, precisely because it is a small package. It makes the build setup slower, especially on ARM. > > 2. It would be useful if it generated the file list automatically, too. > > I had to drop .egg-info and .dist-info manually. I meant: > > I had to drop .egg-info and *add* .dist-info manually. > That, OTOH, sounds like a bug... Nah. It's just the difference between setup.py and pyproject builds. Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
On Sat, Nov 30, 2019 at 05:26:14PM +0100, Dominik 'Rathann' Mierzejewski wrote: > Hi, Miro. > > On Friday, 04 October 2019 at 13:50, Miro Hrončok wrote: > > Hello fellow Python packagers. This is an announcement about a new set > > of RPM macros you can use to build PEP 517/518 enabled packages, that > > is Python packages that have the pyproject.toml file. > > Thanks for your (and whoever else's) work on this. This is exactly the > kind of automation we need to make packaging easier. > > Having said that, I tried to use it in one of my packages (python-gsd) > which started providing the pyproject.toml file in a recent release. > > Two observations: > > 1. It actually generates more BRs than I specify manually (I had 3). It > adds python3-wheel and it brings in python3-pip and python3-pytoml on it > own, so my package ends up with 4 additional BRs for no apparent gain. BR or R? If it just a few additional BR on small python packages, I don't it matters much. > 2. It would be useful if it generated the file list automatically, too. > I had to drop .egg-info and .dist-info manually. That, OTOH, sounds like a bug... Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
Le samedi 30 novembre 2019 à 12:54 -0500, Neal Gompa a écrit : > On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok > wrote: > > On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote: > > > > > > 2. It would be useful if it generated the file list > > > automatically, too. > > > I had to drop .egg-info and .dist-info manually. > > It would. I don't know if we track this somewhere but it is > > certainly on our > > radar. Patrik (CCed) is working on it. > > > > I have done some API drafting and based on RPM limitations, we > > might end up with > > something like this: You have exemples of working implementations in go-rpm-macros and fonts-rpm-macros https://pagure.io/fonts-rpm-macros/ https://pagure.io/go-rpm-macros/ (see rpm/templates in those projects to look how the macros are used) Basically, you need to generate a set of rpm variables that contain at least the target subpackage name, and a way to deduce from it the filelist name (ideally, another clean rpm variable easy to manipulate) Then you just define a macro that generates the %file call for all variable sets. As long a you do it this way it’s fairly easy to manipulate the filelist in %build (or even %install) if the automated process made mistakes. But, generally, a well tested automated generation process will make less errors than a human toiling by rote. Regards, -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
On 30. 11. 19 18:54, Neal Gompa wrote:
On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok wrote:
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
1. It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it
own, so my package ends up with 4 additional BRs for no apparent gain.
python3-pytoml or python3-toml is used to parse the pyproject.toml file.
Unfortunately, while the toml format has been selected for this file, a
standardized python packaging file, there is no toml parser in the stdlib. This
might change in the future some day, but this is what we have now.
Has anyone considered proposing a PEP to mainline a TOML parser
implementation into the standard library yet? We're now nearly two
years into PEP 517 being in force and with Python 2's EOL, I suspect
we'll see the classical setup.py method go away very rapidly. I know
that for my personal projects, the only reason I haven't migrated to
Poetry is because I'm being asked to keep Python 2 compatibility
upstream and dealing with environments running Python 2 makes
pyproject.toml stuff annoying (as they're usually quite old and don't
support it).
Yes and no. See
https://mail.python.org/archives/list/python-id...@python.org/thread/HOZNLCLYPKVCOADNKPTQJDUTNCNZZNZF/#HOKQOMMX2XSS5R2SMXJEHLKXNLW6MVE7
https://github.com/python/peps/issues/1133
2. It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our
radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with
something like this:
%files -n python3-foo -f %{pyproject_files}
Note however that it has a certain drawback. If your "foo" package decides to
suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm,
you will not be able to prevent it via a more strict file rules. That's why I
would also like to offer more "soft" option of this, that would be used somewhat
like this:
%files -n python3-foo -f %{pyproject_metadata}
%{_bindir}/abc
%{python3_sitelib}/xyz/
I.e. it would only contain the dist-info directory. Possibly pycache.
With the dist info directory file list generated (in both concepts), we would be
able to do things like this:
%dir %{python_sitlib}/foo-%{py_version}.dist-info
%license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt
%{python_sitlib}/foo-%{py_version}.dist-info/...
That would however require an upstream way to mark the license file as a license
file. I plan to explore that option early next year.
I'm generally not a fan of the whole generated filelist thing, but
I'll reserve judgement until we have a better idea of what this would
look like.
The general idea I start to like is that the "allowed" (in Fedora) option would
be %{pyproject_metadata} and that would only ever include the dist-info
directory and assertions would be made about it, such as it is only one of them.
Technically all files macro would be possible for fully automated setups, but I
would probably like it banned in regular Fedora.
That is however just my personal view, not an actual plan.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok wrote:
>
> On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
> > 1. It actually generates more BRs than I specify manually (I had 3). It
> > adds python3-wheel and it brings in python3-pip and python3-pytoml on it
> > own, so my package ends up with 4 additional BRs for no apparent gain.
>
> python3-pytoml or python3-toml is used to parse the pyproject.toml file.
> Unfortunately, while the toml format has been selected for this file, a
> standardized python packaging file, there is no toml parser in the stdlib.
> This
> might change in the future some day, but this is what we have now.
>
Has anyone considered proposing a PEP to mainline a TOML parser
implementation into the standard library yet? We're now nearly two
years into PEP 517 being in force and with Python 2's EOL, I suspect
we'll see the classical setup.py method go away very rapidly. I know
that for my personal projects, the only reason I haven't migrated to
Poetry is because I'm being asked to keep Python 2 compatibility
upstream and dealing with environments running Python 2 makes
pyproject.toml stuff annoying (as they're usually quite old and don't
support it).
> Toml parsing is bundled in pip. We could possibly use that instead of an
> additional package, but the API for this is "private" and might change any
> time.
>
> pip itself is used to build the package via pyproject.toml specified build
> backend. We could possibly re-implement everything in the pyproject macros
> project, but given that pip is driving the pyproject.toml standard, we would
> just be in constant need to follow up pip recent development. Not to mention
> NIH. We didnt want to go that way.
>
> Setuptools and wheel is your build backend specified in pyproject.toml (or the
> assumed default). If the upstream chooses to use e.g. flit, sheel would not be
> needed.
>
> Technically, the package gained more dependencies when compared with plain
> "python setup.py build/install". However it is now built differently, more
> close to how things are done in upstream. Frankly speaking the "python
> setup.py
> build/install" is becoming so heavily obsoleted that I suspect modern projects
> to drop setup.py soon-ish.
>
> The entire idea here is that there are more build backends to choose from.
> Hence
> depending on the upstream choices, you'll get:
>
> (py)toml - to parse the files in our macros
> pip - the build frontend
> "whatever" - the build backed
>
> > 2. It would be useful if it generated the file list automatically, too.
> > I had to drop .egg-info and .dist-info manually.
> It would. I don't know if we track this somewhere but it is certainly on our
> radar. Patrik (CCed) is working on it.
>
> I have done some API drafting and based on RPM limitations, we might end up
> with
> something like this:
>
>%files -n python3-foo -f %{pyproject_files}
>
> Note however that it has a certain drawback. If your "foo" package decides to
> suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm,
> you will not be able to prevent it via a more strict file rules. That's why I
> would also like to offer more "soft" option of this, that would be used
> somewhat
> like this:
>
>%files -n python3-foo -f %{pyproject_metadata}
>%{_bindir}/abc
>%{python3_sitelib}/xyz/
>
> I.e. it would only contain the dist-info directory. Possibly pycache.
>
> With the dist info directory file list generated (in both concepts), we would
> be
> able to do things like this:
>
>%dir %{python_sitlib}/foo-%{py_version}.dist-info
>%license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt
>%{python_sitlib}/foo-%{py_version}.dist-info/...
>
> That would however require an upstream way to mark the license file as a
> license
> file. I plan to explore that option early next year.
>
I'm generally not a fan of the whole generated filelist thing, but
I'll reserve judgement until we have a better idea of what this would
look like.
--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
1. It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it
own, so my package ends up with 4 additional BRs for no apparent gain.
python3-pytoml or python3-toml is used to parse the pyproject.toml file.
Unfortunately, while the toml format has been selected for this file, a
standardized python packaging file, there is no toml parser in the stdlib. This
might change in the future some day, but this is what we have now.
Toml parsing is bundled in pip. We could possibly use that instead of an
additional package, but the API for this is "private" and might change any time.
pip itself is used to build the package via pyproject.toml specified build
backend. We could possibly re-implement everything in the pyproject macros
project, but given that pip is driving the pyproject.toml standard, we would
just be in constant need to follow up pip recent development. Not to mention
NIH. We didnt want to go that way.
Setuptools and wheel is your build backend specified in pyproject.toml (or the
assumed default). If the upstream chooses to use e.g. flit, sheel would not be
needed.
Technically, the package gained more dependencies when compared with plain
"python setup.py build/install". However it is now built differently, more
close to how things are done in upstream. Frankly speaking the "python setup.py
build/install" is becoming so heavily obsoleted that I suspect modern projects
to drop setup.py soon-ish.
The entire idea here is that there are more build backends to choose from. Hence
depending on the upstream choices, you'll get:
(py)toml - to parse the files in our macros
pip - the build frontend
"whatever" - the build backed
2. It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our
radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with
something like this:
%files -n python3-foo -f %{pyproject_files}
Note however that it has a certain drawback. If your "foo" package decides to
suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm,
you will not be able to prevent it via a more strict file rules. That's why I
would also like to offer more "soft" option of this, that would be used somewhat
like this:
%files -n python3-foo -f %{pyproject_metadata}
%{_bindir}/abc
%{python3_sitelib}/xyz/
I.e. it would only contain the dist-info directory. Possibly pycache.
With the dist info directory file list generated (in both concepts), we would be
able to do things like this:
%dir %{python_sitlib}/foo-%{py_version}.dist-info
%license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt
%{python_sitlib}/foo-%{py_version}.dist-info/...
That would however require an upstream way to mark the license file as a license
file. I plan to explore that option early next year.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Provisional pyproject RPM macros: Dynamic BuildRequires for Python packages
Hi, Miro. On Friday, 04 October 2019 at 13:50, Miro Hrončok wrote: > Hello fellow Python packagers. This is an announcement about a new set > of RPM macros you can use to build PEP 517/518 enabled packages, that > is Python packages that have the pyproject.toml file. Thanks for your (and whoever else's) work on this. This is exactly the kind of automation we need to make packaging easier. Having said that, I tried to use it in one of my packages (python-gsd) which started providing the pyproject.toml file in a recent release. Two observations: 1. It actually generates more BRs than I specify manually (I had 3). It adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain. 2. It would be useful if it generated the file list automatically, too. I had to drop .egg-info and .dist-info manually. Perhaps I'll try to use the macros in a package with more BRs again. Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
