Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros, and so far was hitting walls left and right :-)
Will look at https://pagure.io/rpkg-util, might have more questions :-) Adam On Tue, Feb 25, 2020 at 12:20 AM clime <cl...@fedoraproject.org> wrote: > Hello! > > On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pin...@pingoured.fr> > wrote: > > > > Good Morning Everyone, > > > > This topic has already been discussed a few times over the past month, > but Adam > > Saleh, Nils Philippsen and myself have had the opportunity to invest > some time > > on it with the hope of making the packager's life simpler as well as > making it > > easier to build automation around our package maintenance. > > > > We have investigated a few ideas, the full list with their pros and cons > can be > > found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev > > If you have any questions about some of these, please ask them, we'll be > happy > > to answer them and potentially complete this document. > > > > > > For both the release and the changelog fields we believe using RPM > macros would > > satisfy the requirements we have for opting-in/out: > > - You can easily opt-in by using the macros > > - You can easily opt-out by removing the macros from your spec file > > > > > > For the changelog, we believe we have a potential good solution: > > - The changelog will be automatically generated using an external > `changelog` > > file as well as the commit history > > - When you opt-in, you will simply move the existing changelog to an > external > > file `changelog`, which is placed in the dist-git repo and add the > > appropriate macro. > > - Upon building, the macro will generate the changelog using all the > commits > > of the repo up to the last commit touching the `changelog` file. Of > all > > these commits it will only consider the commits following these > rules: > > - There are generally two approaches regarding what to include by > default: > > 1. commits touching only the `sources`, `.gitignore`, > `dead.package` > > files, `tests` folder will be ignored by default, i.e. a > blacklist > > 2. only commits touching the spec file or patches will be included > by > > default, i.e. a whitelist > > ==> Which approach do you think is better/will work in most cases? > > - empty commits (not touching any files) will be included on the > assumption > > that this is their purpose > > - commits containing "magic keyword" (#changelog_exclude, > > #changelog_include?) will be ignored or included as the case may be > > - Finally the content of the changelog file specified will be appended > to the > > changelog generated from the git history > > > > > > If you wanted to edit the changelog, you would: > > - Generate the changelog locally via a command like: > > `fedpkg generate-changelog > changelog` > > - Edit `changelog` as desired > > - Commit and push the changes > > > > Since the macro will only consider the commits up to the first commit > touching > > `changelog` (in other words, the macro will only consider the commits > after this > > one) and include `changelog` file as is, your edits will appear in the > RPM > > changelog as you want. > > > > One thing to note is that rebuilds from the same commit will have the > same > > %changelog, they will not get a new entry. If you want a new changelog > entry, > > you have to create a new commit with the desired changelog entry as > commit > > message (the commit itself can be empty). > > > > > > > > However, for the release field, we are struggling a little bit more, two > options > > are more appealing to us: > > > > A) The release field is automatically generated using two elements: > > - the number of commits at this version > > - the number of builds at this commit > > - the dist-tag being added after them > > The release field would thus look like: > > ``<number of commit at version>.<number of build at commit>%{dist}`` > > > > So we could have: (A, B, C and D being different commits) > > A -- version: 0.9 -- release: ? > > | > > B -- version: 1.0 -- release: 1.0 > > | > > C -- version: 1.0 -- release: 2.0 > > | > > D -- version: 1.1 -- release: 1.0 > > > > A rebuild of the commit D, would lead to a release 1.1 (assuming the > first one > > succeeded)/ > > > > Pros: > > - Easy to replicate locally > > - Every changelog entry can be mapped to a `version-release` (No > changes to the > > packaging guidelines) > > - Allows triggering two builds from the same commit without any manual > change > > (simplifies mass-rebuilds) > > - Easy to link a certain build with a specific commit > > - Easy to “guess” the next release value before triggering the build > > > > Cons: > > - Old packages which are no longer receiving upstream releases may need > > special care (for example if they are at the release -48 but only > had 45 > > commits leading to this) > > - Relies on information from the build system for the build number > (but can > > be closely simulated locally since the <n_build> info is only used > for > > rebuilds) > > - Upgrade path may be problematic if Fn is upgraded to version X with > one > > commit while Fn-1 is upgrade to version X with 2 commits (or more) > > > > > > B) The release field is automatically generated taking existing builds > in all > > current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This > means for > > builds of the same epoch:version, find a new release that (if at all > possible) > > ensures upgradability from older Fedora versions to newer ones, adhering > to the > > structure of a release tag documented in the Versioning Guidelines[1]. > Going > > from the (RPM-wise) "latest build" that the new one should surpass, this > can > > mean bumping in the front (`pkgrel`) or the back (`minorbump`). > > > > This allows packages from "very stable" upstreams who haven't released > in years > > to still benefit from automatically generated releases. > > > > The following examples would use a macro for the release field as > outlined in a > > separate document[2]. > > > > Example #1 ("normal" release progression): > > Rawhide has: 2.0-1.fc32 > > F31 has: 1.0-1.fc31 > > F30 has: 1.0-1.fc30 > > > > Next release in F31: 1.0-2.fc32 > > > > > > Example #2 ("hotfix" in an older release, selected by an alternative > macro (or > > option) in the spec file): > > Rawhide has: 2.0-1.fc32 > > F31 has: 1.0-1.fc31 > > F30 has: 1.0-1.fc30 > > > > Next release in F30: 1.0-1.fc30.1 > > > > Example #3 (pre-release, selected by an alternative macro (or option) in > the > > spec file): > > Rawhide has: 2.0-1.fc32 > > F31 has: 1.0-1.fc31 > > F30 has: 1.0-1.fc30 > > > > Next release in Rawhide: 3.0-0.1.20200224git1234abcd > > > > > > Pros: > > - Allows triggering two builds from the same commit without manual > > intervention > > - Emulates what many maintainers do manually today for most use cases > > - Can cater for pre-releases (e.g.: by offering different macros or > macro > > options for the different use cases) > > > > Cons: > > - Needs the build system for information about builds in this and > other Fedora > > versions > > - Not easy to reproduce locally because we don’t have > machine-consumable > > knowledge about other builds in e.g. the dist-git repo > > - Does not allow to generate changelog entries with `version-release` > > information for all commits (and this will require a change in our > packaging > > guidelines) > > > > > > So these are the results of our current investigations, we are very much > eager > > to get your feedback on them and even more eager if you have ideas on > how to > > approach/solve some of the challenges mentioned here. > > > > > > Looking forward for the discussion, > > > > Pierre > > For Adam, Nils and myself > > What is the point of including number of builds into release? I think > the Miro's approach solves it. > Or is there any other problem except soname bumps? > > Ad. document - annotated git tags: > (-) Editing the changelog would mean allowing to remove the git tags, > thus leading to potential issue in build reproducibility > > That doesn't need to be the case. In rpkg-util, this was resolved by > introducing arguments since_tag and until_tag > for git_changelog macro > (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). > That's > how it can be prevented for some annotated tag to contribute to changelog. > > Example: > > {{{ git_changelog since_tag=name-1.3-1 }}} > > * Mon Feb 24 2020 clime <cl...@fedoraproject.org> 1.2-1 > - manual changelog entry that is used instead of a tag annotation > > {{{ git_changelog until_tag=name-1.1-1 }}} > > Removing already pushed git tags is probably not a good idea anyway: > https://git-scm.com/docs/git-tag#_on_re_tagging > > Ad. the following approach for calculating release: > - Compute the release field from the number of commits since the last > version change: <version>-<commits_number>%{dist} > > I think it is a good idea. In rpkg-util, it is done similarly, except > the release part has more subparts than just > two (including %{dist}). > > The full description is here: > https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 > but the main difference is that it counts commits from the latest > annotated tag which contains (in its name) > the current version and the current release number which are extracted > from the spec file when > creating the tag unless they are specified manually on command line. > Commit count is only appended > to it if we build from commit which is on top of some annotated tag > (i.e. it is itself untagged). > > Going by just a textual version change in a spec file might be slightly > tricky. > You would need to go through all the past commits that touched that spec > file, > keep checking these out and look if the version is changed when compared > to the > one parsed from the head commit. Possible though. > > To go back to your original post: > > For both the release and the changelog fields we believe using RPM > macros would > > satisfy the requirements we have for opting-in/out: > > By using such RPM macros, you would lose ability to rebuild srpms > because it will > be only possible to build them when the git context is present but not > when they > become a standalone thing. I.e. things like this will not work: > > > https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mockbuild/backend.py#L270 > > That's why I think that such macros should be of a different kind: > macros that are computed > once when srpm is created and result of which is put _verbatim_ into > the spec file so that > there is no (re)computation later when srpm is being built and when > the git context is already > missing. > > This approach is taken in the rpkg-util project > (https://pagure.io/rpkg-util). I really > recommend looking at it as I spent more than a year solving this > particular problem with > changelog and release (but actually not only that problem) and I also > optimized the macros there > as much as I possibly could. > > If you want to play around with it, you can download the latest > version from here: > > https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ > > and try it on this: > > https://pagure.io/hello_rpkg > > clime > > > > > > > [1]: > https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/ > > [2]: https://hackmd.io/kuXOPe74RfepuztBz7pBsg > > _______________________________________________ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org >
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
