Re: Automatic Provides: Discussion summary and plan

[Jason L Tibbitts III]

> "PV" == Petr Viktorin  writes:

PV> - Make it standard practice in Fedora to use this data and treat the
PV> spec file as an immutable generated artifact.

If you're saying that any changes which are made to the spec file (say,
by release engineering doing a rebuild or by someone tweaking the
package for pretty much any reason) will be overwritten, this is not a
particularly good idea.

 - J<
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Petr Viktorin]

On 08/17/2016 06:52 PM, Nick Coghlan wrote:

On 15 August 2016 at 19:42, Igor Gnatenko  wrote:

It can't track/change BR/R's as RPM is Turing complete and impossible to parse.
Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in
Fedora we still need to add some more BR for that, so we add it. When
new release comes (still without added BR in upstream) rebase-helper
will not be helpful. It can change only version of spec.


This is a self-inflicted problem arising from our current tooling
design, though, since "generate-and-edit" isn't the only way to
supplement upstream metadata: we can also design spec file generators
to accept a supplemental config file in addition to the upstream
metadata.

If we're using that alternate model, then rebase-helper can have a
much easier time of things, since it isn't trying to edit a previously
generated spec file, it's just generating a new one based on the new
upstream metadata and the old supplemental metadata, and seeing if the
result still passes CI.


The more I think about it, the more it seems to me that our plan should be:
- Make it possible for pyp2rpm to use extra data to augment/override 
what's in the upstream package.
- Make it standard practice in Fedora to use this data and treat the 
spec file as an immutable generated artifact.
- Treat metadata errors/omissions as upstream bugs, provided the desired 
state can be expressed in setup.py
- For kinds of metadata that setup.py can't express, strive to add them 
to future versions of the upstream packaging format.
- For the far future, perhaps start getting rid of spec files: teach 
Koji to generate them, and stop storing them in dist-git.



Helper macros stay orthogonal -- pyp2rpm would just need to learn to use 
them.



--
Petr Viktorin
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Nick Coghlan]

On 15 August 2016 at 19:42, Igor Gnatenko  wrote:
> It can't track/change BR/R's as RPM is Turing complete and impossible to 
> parse.
> Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in
> Fedora we still need to add some more BR for that, so we add it. When
> new release comes (still without added BR in upstream) rebase-helper
> will not be helpful. It can change only version of spec.

This is a self-inflicted problem arising from our current tooling
design, though, since "generate-and-edit" isn't the only way to
supplement upstream metadata: we can also design spec file generators
to accept a supplemental config file in addition to the upstream
metadata.

If we're using that alternate model, then rebase-helper can have a
much easier time of things, since it isn't trying to edit a previously
generated spec file, it's just generating a new one based on the new
upstream metadata and the old supplemental metadata, and seeing if the
result still passes CI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Igor Gnatenko]

It can't track/change BR/R's as RPM is Turing complete and impossible to parse.
Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in
Fedora we still need to add some more BR for that, so we add it. When
new release comes (still without added BR in upstream) rebase-helper
will not be helpful. It can change only version of spec.

On Mon, Aug 15, 2016 at 11:25 AM, Tomas Orsava  wrote:
> On 08/12/2016 05:40 PM, Petr Viktorin wrote:
>>
>> The idea with pyp2rpm is to work with the Python Packaging Authority so
>> that the upstream metadata *can* be converted automatically. Ideally Fedora
>> packagers would fix packaging problems upstream, rather than maintaining
>> spec files.
>> Ideas for a tool for *updating* spec files are also floating around.
>>
> There's already rebase-helper (it's an interactive tool as well as an
> automatic one) that specializes in updating spec files.
>
> ___
> python-devel mailing list
> python-devel@lists.fedoraproject.org
> https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org



-- 
-Igor Gnatenko
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Petr Viktorin]

On 08/12/2016 03:17 PM, Jason L Tibbitts III wrote:

"PV" == Petr Viktorin  writes:


PV> The magic worries me. It seems like if these macros were finished,
PV> you'd be about the only person capable of maintaining them.

I don't think so.  There are other committers, and the core of it didn't
even come from me so there's certainly at least two other people around
who can handle this stuff.  It's true that not a whole lot of people
understand the deeper interactions between the RPM Lua interpreter and
the rest of RPM, but it took me only about two days to figure out most
of it _without_ having any documented examples.  And I have a real job
that isn't Fedora.


Thanks for clearing that up, I'm less worried now.


The macros are very well commented; the magic is described in detail.
They also include a debugging framework.  Which is more than, well, any
other current macro magic.  Ever tried to debug debuginfo package
generation?  Or even looked at the SCL macros?  (Which are about on par
with magic-ness, but which are completely unreadable and have no
debugging.)


The macros are definitely of exceptional quality. But, still they're RPM 
macros.
(Yes, I have looked SCL macros – that's something I'd not encourage 
people to study...)



PV> And if something goes wrong (magic tends to imply fragility), I'm
PV> not looking forward to the debugging sessions. So, while I am blown
PV> away by this project, I'm inclined to place my bets on pyp2rpm
PV> instead.

Well, there's no reason not to have more than one solution.  However,
pyp2rpm just gets you a spec.  You still have to maintain said spec, and
wiping it out with a fresh run of the generator is not really
acceptable.  I'd generally argue that pyp2rpm would actually generate a
spec using this stuff, once it's actually proven that it works.


The idea with pyp2rpm is to work with the Python Packaging Authority so 
that the upstream metadata *can* be converted automatically. Ideally 
Fedora packagers would fix packaging problems upstream, rather than 
maintaining spec files.

Ideas for a tool for *updating* spec files are also floating around.

The idea of pyp2rpm generating spec files using the macros sounds good. 
Indeed the projects are more orthogonal than I realized.



I would still like to have the spec file be simply an intermediary
between some metadata and the build system.  Or, if you can write
pyp2rpm in lua, I could actually build it directly into rpm.  Have the
regular RPM header, then a %prep section that unpacks the source and
calls a macro.  The rest of the spec (except the %changelog section) is
simply generated from the metadata.


Agreed; it would be nice to get to that kind of situation in the future. 
Not sure about the lua part, at least while pyp2rpm needs heavy 
development :)


--
Petr Viktorin
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Petr Viktorin]

On 08/11/2016 07:37 PM, Jason L Tibbitts III wrote:

"TO" == Tomas Orsava  writes:


TO> That looks incredible! Why didn't it see the light of day? Time
TO> constraints or some technical issues?

Well, it sort of fell by the wayside as I got involved with other
things.  I've learned a lot about RPM internals since then and I know I
really should get back to it.  So many things higher up in the
(incredibly huge) queue.

TO> Maybe it could be revived?

Technically it isn't dead; I just haven't worked on it in a (long)
while.

Basically I tried to come up with the ideal spec file and worked
backwards from that.  It's still not going to work for every package,
and it still isn't remotely as nice as simply not using an RPM spec at
all (and just generating one from metadata) but I think it's at least
a start.

Also, it does horrible, horrible things behind the scenes because RPM
just doesn't give us a couple of needed bits.  I filed a ticket with RPM
upstream to try and have it do that but no luck.  But the actual macros
behind the scenes are very well commented so at least things should be
moderately obvious.


The magic worries me. It seems like if these macros were finished, you'd 
be about the only person capable of maintaining them. And if something 
goes wrong (magic tends to imply fragility), I'm not looking forward to 
the debugging sessions. So, while I am blown away by this project, I'm 
inclined to place my bets on pyp2rpm instead.


I don't think the end goals – not having to write a spec at all, or 
write an ideal spec – are as important as the debugging experience.


But, that's all just my view; I have no intentions of hindering the 
project, and I encourage anyone involved with RPM macros to study it and 
see what's possible.



--
Petr Viktorin
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Jason L Tibbitts III]

> "TO" == Tomas Orsava  writes:

TO> That looks incredible! Why didn't it see the light of day? Time
TO> constraints or some technical issues?

Well, it sort of fell by the wayside as I got involved with other
things.  I've learned a lot about RPM internals since then and I know I
really should get back to it.  So many things higher up in the
(incredibly huge) queue.

TO> Maybe it could be revived?

Technically it isn't dead; I just haven't worked on it in a (long)
while.

Basically I tried to come up with the ideal spec file and worked
backwards from that.  It's still not going to work for every package,
and it still isn't remotely as nice as simply not using an RPM spec at
all (and just generating one from metadata) but I think it's at least
a start.

Also, it does horrible, horrible things behind the scenes because RPM
just doesn't give us a couple of needed bits.  I filed a ticket with RPM
upstream to try and have it do that but no luck.  But the actual macros
behind the scenes are very well commented so at least things should be
moderately obvious.

Anyway, I'll try to have another look at it at some point.  I need to
extract the debug scaffolding and such and put that into Fedora
separately so that I can avoid rewriting it all the time.

 - J<
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Tomas Orsava]

Yeah, I really wish I had actually pushed through the macro work I had
done last year.  You can see that at https://pagure.io/python-macros

A spec would look like this:
https://pagure.io/python-macros/blob/master/f/test1.spec

And most of that is actually implemented.  Note the almost complete
absence of version-specific bits.  That package builds for an arbitrary
number of system python versions automatically.

That looks incredible! Why didn't it see the light of day? Time 
constraints or some technical issues?


Maybe it could be revived?

Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Jason L Tibbitts III]

> "PV" == Petr Viktorin  writes:

PV> No, getting. Example expansion:
PV> %{python_dist_name ndg_HTTPSclient} => ndg-httpsclient

Ah, OK, that makes much more sense.

PV> Would "python3_requires" and "python3_buildrequires" be better?

I think so.

PV> Please do and share your thoughts, I trust you have a good feel for
PV> consistency here.

Yeah, I really wish I had actually pushed through the macro work I had
done last year.  You can see that at https://pagure.io/python-macros

A spec would look like this:
https://pagure.io/python-macros/blob/master/f/test1.spec

And most of that is actually implemented.  Note the almost complete
absence of version-specific bits.  That package builds for an arbitrary
number of system python versions automatically.

 - J<
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Jason L Tibbitts III]

> "PV" == Petr Viktorin  writes:

PV> * One macro for getting the canonical (normalized) dist-name:
PV> %{python_dist_name NAME}

Do you mean "setting" here?


PV> * Four macros for adding Requires and BuildRequires lines (which use
PV>   the python_dist_name macro above):

PV>   %{requires_python3_dist NAME} => Requires: python3.Ydist(name)
PV>   %{buildrequires_python3_dist NAME} => BuildRequires: %python3dist(name)
PV>   and similarly for Python 2.

Hmm, I'll have to think about this.  I do prefer the python macros all
actually start with "python" or "py" just to make them easily
recognizable and memorable.  And making them too long, well, just
contributes to verbosity which we'd like to get away from.

I also need to think about how this fits in with other macro work that's
been happening.

 - J<
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Automatic Provides: Discussion summary and plan

[Petr Viktorin]

To help automatic building RPMs from native Python packages, we need an
automatic way to record the software's upstream name (dist name, name
on PyPI) in Fedora packages.

For this, we are using RPM's automatic dependency generator written by
Per Øyvind Karlsen and Neal Gompa, which automatically scans
".dist-info" and similar files and generates virtual Provides,
currently in the form "pythonX.Ydist(name)".
(The generator was originally developed for Mandriva and Mageia; finally
it's coming to Fedora as well!)
The Fedora Change page for this feature is here:
https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages

Unfortunately, while implementing this, we failed to understand how the
virtual provides work with source RPMs. This leads to the situation
that, as currently planned, the automatic provides are only useful in
"Requires" lines, not in "BuildRequires".

The problem is that if a requirement like "python3.5dist(name)" is
present in a SRPM, the SRPM can't be rebuilt on systems with other
Python versions, which would provide, say, "python3.6dist(name)".
For these systems, the SRPM would need to be rebuilt, which not all
tools do.

Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python,
after which we had a long discussion about the problem and possible
solutions. We're sorry for not getting this earlier -- despite several
people mentioning it (including Neal who wrote the generator)!


The fix is to enable "--majorver-provides" in the dependency generator,
so that each package will provide two forms of the virtual provide:
python3.5dist(name)
python3dist(name)
The full version should then be used for runtime Requires, while the
one with major version only should be used for BuildRequires.

We'll wrap this up in easy-to use (and hopefully future-proof) macros:

* One macro for getting the canonical (normalized) dist-name:
%{python_dist_name NAME}

* Four macros for adding Requires and BuildRequires lines (which use the
  python_dist_name macro above):

  %{requires_python3_dist NAME} => Requires: python3.Ydist(name)
  %{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name)
  and similarly for Python 2.

An alternative would be macros that don't include the keyword "Requires:" or
"BuildRequires:". This would result in specfiles with lines like:
BuildRequires: %{python3_dist_br name}
Requires: %{python3_dist name}
This would be susceptible to people copying the Requires line, adding
"Build" to the front, and forgetting to change the macro as well,
leading to a hard-to-catch failure (working binary RPMs but SRPMs that
fail to rebuild on other distro versions).
Macros that include the keyword are more robust to copy-pasting.

Since the %buildrequires_python3_dist and %python_dist_name macros will
be used in the SRPM, they'll need to go in macros.python-srpm to be
present in the default buildroot.
The %requires_python3_dist macro (and %pythonX_version) can live in
macros.pythonX.


Following is the new road plan:


# Plan for Fedora 25:
* The 5 macros will be created and deployed.
* The --majorver-provides swich for the Provides generator in Fedora RPM 
will

  be enabled
* Fedora Packaging Guidelines for Python will be amended:
* The addition of the pythonX.Ydist(..) tags will be explained.
* The %{python_dist_name} macro will be advertised.
* The %{requires_pythonX_dist} macros will be advertised for use in
  generating Requires tags.
* It will be explained that, at this time, it is impossible to
  generate BuildRequires without the providing package being
  rebuilt, so the %{buildrequires_pythonX_dist} macros will be
  discouraged for now.
* See if we can get in another targeted mass rebuild. If yes, continue
  with the f26 plan early.


# Plan for Fedora 26:
* All Provides will be regenerated in the regular Fedora 26 mass rebuild
* Change Python guidelines so the %{buildrequires_pythonX_dist} macros
  are now encouraged.


--
Petr Viktorin
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org