Re: [Fedora-packaging] Re: Ideas and proposal for removing changelog and release fields from spec file
Le 2020-03-03 15:14, clime a écrit :
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot
wrote:
Le 2020-03-02 14:45, clime a écrit :
> On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel
> wrote:
>>
>> Le 2020-03-01 02:31, clime a écrit :
>> > On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel
>> > wrote:
>> >>
>> >> Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
>>
>> >> Putting %{dynrel} reconciliation in the rpmbuild -bs stage using
>> >> detached file state means that fedpkg local (or checking out git state
>> >> and building in mock or copr or OBS or via plain rpmbuild -bs) will
>> >> give you the same result as launching fedpkg build.
>> >
>> > Well, I believe it doesn't. You run:
>> >
>> > 1) fedpkg local -> produces -1.0-1.fc32 (1 in release because
>> > you haven't built that package before)
>> > 2) fedpkg build
>>
>> At that point state is undefined till the build succeeds or not. If
>> the
>> build succeeds, the buildsystem will write back a new state. Let’s
>> assume it succeeds.
>
> It's undefined if you add more elements to the equation than necessary
> (i.e. build system), otherwise it would be well-defined in this case.
>
>>
>> > 3) vim .spec and do some change in %description
>> > 4) fedpkg commit -m "description improvement"
>> > 5) fedpkg local -> produces -1.0-1.fc32
>> > 6) fedpkg push -> error because build system pushed meanwhile
>>
>> Yes, here the packager notices something else has been going on, and
>> he
>> needs to merge or rebase. He’d have the same effect if another
>> packager
>> had been doing stuff on the package, or a mass rebuild had been going
>> on. That’s the distributed decentralized aspect of git, except here
>> the
>> packager collided with himself by starting two work threads in
>> parallel
>> (one buildsys-side, one local).
>
> The problem is that you launched some process and you need to wait
> until it finishes, normally you don't need to anything like that when
> working with git.
But fedpkg build is a release to production process. It’s not a dev
staging process. In release processes, actually doing the release is
not
an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately
continue working after making a release. Probably rare but could
happen and if build time of a package is long (libreOffice/firefox),
this could hit you as an inconvenience. Or do you want to push-back
immediately after srpm build when you don't yet know whether the build
will succeed? That wasn't clear to me before.
Either you lock the centralized branch while the build is ongoing or you
fail the build if someone pushes to the centralized branch before the
end of the build.
That does not stop packagers from preparing the next stage in a local
branch, only changing the state of the centralized branch at the same
time the build process is changing it (two conflicting release state
changes, option 1: buildsystem wins, option 2, packager wins, can’t have
an heisenstate where things are both released and not released).
I am not suggesting to use raw git commit messages for the changelog
but instead content of annotated tags which can be initially
prepopulated by commit messages but can be then edited to any extent
and even the way the edit window for annotated tag is prepopulated
might be configurable (to .e.g. load content from a file in which case
you would probably skip the editing part). And I see the annotated
tags as the actual releases.
The only actual release is the package built in koji. So, no matter what
mechanism you use git side:
– you must represent the real state of package builds and not a git-only
fiction
→ that means a form of buildsys write-back because builds can and will
fail
– you must produce something packagers can also build in mock or copr or
OBS or whatever
→ that means checkout-ing changelog data in a form that can be read by
rpmbuild -bs
→ ie a file not git metadata
→ with the associated conflict risk if multiple changes occur between
checkouts
You can not avoid a buildsys merge-back when doing production builds.
Merge backs by build system can be avoided, however. Why do you think
they can't be avoided?
Because they’re the real state rpm changelogs records. Because that’s
what people use rpm changelogs for. Something broke, what is the
affected packaged, what where the released package states since last
time it worked.
There is the case of mass rebuilds but this is pretty much a one-shot
event
Mass rebuilds are not an exception, they’re becoming the norm. Every SIG
that deals with modern software released as a huge number of interlinked
components needs to perform SIG-level mass rebuilds all the time
(directly in rawhide, in a side tag, whatever). koshei can now autobuild
all dependent packages when one of their requirements changed. That’s
where the project is going and has been going for several years.
All the modularity efforts in the past years
Re: [Fedora-packaging] Re: Ideas and proposal for removing changelog and release fields from spec file
On Tue, 3 Mar 2020 at 09:38, Nicolas Mailhot
wrote:
>
> Le 2020-03-03 07:03, clime a écrit :
>
> > Actually, that wouldn't work because prefix needs to be static, not
> > dependent on rpm macros
>
> For myself, I would oppose any rpm release process that would not take
> core rpm mecanisms like macros into account.
Being independent of rpm has one huge potential advantage. You can
apply the same approach that you apply for rpm spec files to other
kinds of spec files.
For example the fedora & containers mailing list contains a thread,
when person needs to parametrize FROM clause by branch name to
correctly reference base image:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/65GHZM4PODSO36C66ONZCYXVIN7JIXMK/
The approach I am suggesting would help there and it could help on
other places too.
>
> Recording builds in changelogs without checking they actually happened
> is bad engineering.
It's not about recording builds but instead about recording releases
which I see as something that takes place already in Git.
>
> Simulating rpm behaviour without performing actual spec evaluation in
> rpm, is also bad engineering. Especially when you know your simulation
> is horribly simplistic and approximative.
I am not trying to simulate rpm behavior. I am just trying to
introduce macros that are rendered verbatim into the spec file inside
srpm so those srpms can be rebuilt at any time when needed. rpm does
not have this feature atm. If we should put it into rpm, it would be
already third kind of rpm macro but even if it should go into rpm at
some point, it is better to have the approach heavily tested by
production use first because here it doesn't really add much
complexity. And as I mentioned, independence of rpm means you can use
it for non-rpm projects too.
It's also not simplistic: what you see in {{{ }}} tags is interpreted
by bash, it can be multi-line and you can hence do anything there you
can do with rpm macros. The only problem is, people already have some
automation in rpm macros that doesn't require the git context, instead
just e.g. the tarball content and this functionality would need to be
rewritten by using {{{ }}} if it should play together with the
git-based release generation. This cannot be avoided as much as I
would like to. If we try to avoid it, we will get a weird hybrid that
doesn't work.
It's also not approximative. You can define your own logic for
generating release. There should be some pre-made recommended macros
but in the end you are not limited by them.
>
> “Who cares if results match most of the time” is terrible workload
> optimization. You’ll make packagers waste far more time fixing the cases
> where automation guessed wrong, than you will win when it guesses right.
> Basic 80/20 rule, the 20% hard cases cost more than the 80% easy cases.
> Taking care of the 80% easy cases while making the 20% hard cases harder
> (due to automation mistakes) is not a good deal at all.
>
> Please work on approaches which are reliable by default. Reliability is
> hard even when you aim for it. When you don’t, it’s not attainable at
> all.
I think the approach I am suggesting is very reliable.
As much as I enjoy this conversation, I will need to actually try to
make something happen. Forgive me if I don't go into long disputes
hereafter. If you, however, make a detailed document summarizing the
infra changes needed for your approach I will try to comment on the
individual aspects of it and also respond with the document with the
changes needed for what I am suggesting.
Best regards!
clime
>
> Reagrds,
>
> --
> Nicolas Mailhot
> ___
> packaging mailing list -- packag...@lists.fedoraproject.org
> To unsubscribe send an email to packaging-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: Ideas and proposal for removing changelog and release fields from spec file
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot
wrote:
>
> Le 2020-03-02 14:45, clime a écrit :
> > On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel
> > wrote:
> >>
> >> Le 2020-03-01 02:31, clime a écrit :
> >> > On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel
> >> > wrote:
> >> >>
> >> >> Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
> >>
> >> >> Putting %{dynrel} reconciliation in the rpmbuild -bs stage using
> >> >> detached file state means that fedpkg local (or checking out git state
> >> >> and building in mock or copr or OBS or via plain rpmbuild -bs) will
> >> >> give you the same result as launching fedpkg build.
> >> >
> >> > Well, I believe it doesn't. You run:
> >> >
> >> > 1) fedpkg local -> produces -1.0-1.fc32 (1 in release because
> >> > you haven't built that package before)
> >> > 2) fedpkg build
> >>
> >> At that point state is undefined till the build succeeds or not. If
> >> the
> >> build succeeds, the buildsystem will write back a new state. Let’s
> >> assume it succeeds.
> >
> > It's undefined if you add more elements to the equation than necessary
> > (i.e. build system), otherwise it would be well-defined in this case.
> >
> >>
> >> > 3) vim .spec and do some change in %description
> >> > 4) fedpkg commit -m "description improvement"
> >> > 5) fedpkg local -> produces -1.0-1.fc32
> >> > 6) fedpkg push -> error because build system pushed meanwhile
> >>
> >> Yes, here the packager notices something else has been going on, and
> >> he
> >> needs to merge or rebase. He’d have the same effect if another
> >> packager
> >> had been doing stuff on the package, or a mass rebuild had been going
> >> on. That’s the distributed decentralized aspect of git, except here
> >> the
> >> packager collided with himself by starting two work threads in
> >> parallel
> >> (one buildsys-side, one local).
> >
> > The problem is that you launched some process and you need to wait
> > until it finishes, normally you don't need to anything like that when
> > working with git.
>
> But fedpkg build is a release to production process. It’s not a dev
> staging process. In release processes, actually doing the release is not
> an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately
continue working after making a release. Probably rare but could
happen and if build time of a package is long (libreOffice/firefox),
this could hit you as an inconvenience. Or do you want to push-back
immediately after srpm build when you don't yet know whether the build
will succeed? That wasn't clear to me before.
>
> You're asked to autobump a “Release” field. You’re asked to automate a
> changelog which is effectively a release changelog, not dev changelog
> (the dev changelog lives in git, people want a good changelog in rpm
> proper to track release not git history).
I am not suggesting to use raw git commit messages for the changelog
but instead content of annotated tags which can be initially
prepopulated by commit messages but can be then edited to any extent
and even the way the edit window for annotated tag is prepopulated
might be configurable (to .e.g. load content from a file in which case
you would probably skip the editing part). And I see the annotated
tags as the actual releases.
From https://git-scm.com/docs/git-tag:
Annotated tags are meant for release while lightweight tags are meant
for private or temporary object labels.
They were designed to represent releases and because we are using git
I don't see why we should avoid using annotated tags for releases.
On src.fp.o, there is also already support for this: e.g.
https://src.fedoraproject.org/rpms/2ping/releases
The release needs building (koji), and distribution (bodhi + mirrors)
but that's stuff that needs to be done with the release. Scratch
builds can be used to make sure that the package builds and in the
worst case, a new release needs to be done that accounts for changed
build environment. But I don't think we need to throw the whole
concept of annotated tags = releases because of it.
Of course, everything here can be made opt-in. As an approach somebody
likes and somebody doesn't.
I think this is where our approaches really differ, you see release as
something that happens somewhere else than in git, but I see the
release as something that happens in git. And afterwards we need to
get that released content to users.
I think we will never be able to reconciliate our views because they
are different from the very base. In the end, what matters is
implementation complexity and ease of use. I can present to you the
things that need to be done for this to work in a detailed way
(although it would take a bit of time to put into a document) but I
also know it is a doable and it is actually fairly simple when not
accounting for time spending on pull requests and getting them
accepted. Your approach seems to be intuitively difficult to
implement, either locking implementation (i am
