On Tue, 25 Feb 2020 at 09:31, clime <cl...@fedoraproject.org> wrote: > > Hello Adam! > > On Tue, 25 Feb 2020 at 08:58, Adam Saleh <asa...@redhat.com> wrote: > > > > Nice, I have been trying to fight through the 'git context already missing' > > with pure lua rpm macros, > > and so far was hitting walls left and right :-) > > > > Will look at https://pagure.io/rpkg-util, might have more questions :-) > > You have probably already noticed...the docs at > https://docs.pagure.org/rpkg-util/index.html > are for rpkg-util version that is currently in Fedora but it contains > some nice introduction, nevertheless. > > The (newer) git version from the pagure repository has some features > (e.g. git_release macro), > which are not yet documented there. > > man page of the git version ($ man rpkg) is up to date, however. > > Thank you for looking at it. > clime
...in case of any questions i am on this email or on #fedora-apps. Again, thanks for looking at it. Cheers. > > > > > Adam > > > > On Tue, Feb 25, 2020 at 12:20 AM clime <cl...@fedoraproject.org> wrote: > >> > >> Hello! > >> > >> On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pin...@pingoured.fr> > >> wrote: > >> > > >> > Good Morning Everyone, > >> > > >> > This topic has already been discussed a few times over the past month, > >> > but Adam > >> > Saleh, Nils Philippsen and myself have had the opportunity to invest > >> > some time > >> > on it with the hope of making the packager's life simpler as well as > >> > making it > >> > easier to build automation around our package maintenance. > >> > > >> > We have investigated a few ideas, the full list with their pros and cons > >> > can be > >> > found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev > >> > If you have any questions about some of these, please ask them, we'll be > >> > happy > >> > to answer them and potentially complete this document. > >> > > >> > > >> > For both the release and the changelog fields we believe using RPM > >> > macros would > >> > satisfy the requirements we have for opting-in/out: > >> > - You can easily opt-in by using the macros > >> > - You can easily opt-out by removing the macros from your spec file > >> > > >> > > >> > For the changelog, we believe we have a potential good solution: > >> > - The changelog will be automatically generated using an external > >> > `changelog` > >> > file as well as the commit history > >> > - When you opt-in, you will simply move the existing changelog to an > >> > external > >> > file `changelog`, which is placed in the dist-git repo and add the > >> > appropriate macro. > >> > - Upon building, the macro will generate the changelog using all the > >> > commits > >> > of the repo up to the last commit touching the `changelog` file. Of > >> > all > >> > these commits it will only consider the commits following these > >> > rules: > >> > - There are generally two approaches regarding what to include by > >> > default: > >> > 1. commits touching only the `sources`, `.gitignore`, > >> > `dead.package` > >> > files, `tests` folder will be ignored by default, i.e. a > >> > blacklist > >> > 2. only commits touching the spec file or patches will be included > >> > by > >> > default, i.e. a whitelist > >> > ==> Which approach do you think is better/will work in most cases? > >> > - empty commits (not touching any files) will be included on the > >> > assumption > >> > that this is their purpose > >> > - commits containing "magic keyword" (#changelog_exclude, > >> > #changelog_include?) will be ignored or included as the case may be > >> > - Finally the content of the changelog file specified will be appended > >> > to the > >> > changelog generated from the git history > >> > > >> > > >> > If you wanted to edit the changelog, you would: > >> > - Generate the changelog locally via a command like: > >> > `fedpkg generate-changelog > changelog` > >> > - Edit `changelog` as desired > >> > - Commit and push the changes > >> > > >> > Since the macro will only consider the commits up to the first commit > >> > touching > >> > `changelog` (in other words, the macro will only consider the commits > >> > after this > >> > one) and include `changelog` file as is, your edits will appear in the > >> > RPM > >> > changelog as you want. > >> > > >> > One thing to note is that rebuilds from the same commit will have the > >> > same > >> > %changelog, they will not get a new entry. If you want a new changelog > >> > entry, > >> > you have to create a new commit with the desired changelog entry as > >> > commit > >> > message (the commit itself can be empty). > >> > > >> > > >> > > >> > However, for the release field, we are struggling a little bit more, two > >> > options > >> > are more appealing to us: > >> > > >> > A) The release field is automatically generated using two elements: > >> > - the number of commits at this version > >> > - the number of builds at this commit > >> > - the dist-tag being added after them > >> > The release field would thus look like: > >> > ``<number of commit at version>.<number of build at commit>%{dist}`` > >> > > >> > So we could have: (A, B, C and D being different commits) > >> > A -- version: 0.9 -- release: ? > >> > | > >> > B -- version: 1.0 -- release: 1.0 > >> > | > >> > C -- version: 1.0 -- release: 2.0 > >> > | > >> > D -- version: 1.1 -- release: 1.0 > >> > > >> > A rebuild of the commit D, would lead to a release 1.1 (assuming the > >> > first one > >> > succeeded)/ > >> > > >> > Pros: > >> > - Easy to replicate locally > >> > - Every changelog entry can be mapped to a `version-release` (No > >> > changes to the > >> > packaging guidelines) > >> > - Allows triggering two builds from the same commit without any manual > >> > change > >> > (simplifies mass-rebuilds) > >> > - Easy to link a certain build with a specific commit > >> > - Easy to “guess” the next release value before triggering the build > >> > > >> > Cons: > >> > - Old packages which are no longer receiving upstream releases may need > >> > special care (for example if they are at the release -48 but only > >> > had 45 > >> > commits leading to this) > >> > - Relies on information from the build system for the build number > >> > (but can > >> > be closely simulated locally since the <n_build> info is only used > >> > for > >> > rebuilds) > >> > - Upgrade path may be problematic if Fn is upgraded to version X with > >> > one > >> > commit while Fn-1 is upgrade to version X with 2 commits (or more) > >> > > >> > > >> > B) The release field is automatically generated taking existing builds > >> > in all > >> > current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This > >> > means for > >> > builds of the same epoch:version, find a new release that (if at all > >> > possible) > >> > ensures upgradability from older Fedora versions to newer ones, adhering > >> > to the > >> > structure of a release tag documented in the Versioning Guidelines[1]. > >> > Going > >> > from the (RPM-wise) "latest build" that the new one should surpass, this > >> > can > >> > mean bumping in the front (`pkgrel`) or the back (`minorbump`). > >> > > >> > This allows packages from "very stable" upstreams who haven't released > >> > in years > >> > to still benefit from automatically generated releases. > >> > > >> > The following examples would use a macro for the release field as > >> > outlined in a > >> > separate document[2]. > >> > > >> > Example #1 ("normal" release progression): > >> > Rawhide has: 2.0-1.fc32 > >> > F31 has: 1.0-1.fc31 > >> > F30 has: 1.0-1.fc30 > >> > > >> > Next release in F31: 1.0-2.fc32 > >> > > >> > > >> > Example #2 ("hotfix" in an older release, selected by an alternative > >> > macro (or > >> > option) in the spec file): > >> > Rawhide has: 2.0-1.fc32 > >> > F31 has: 1.0-1.fc31 > >> > F30 has: 1.0-1.fc30 > >> > > >> > Next release in F30: 1.0-1.fc30.1 > >> > > >> > Example #3 (pre-release, selected by an alternative macro (or option) in > >> > the > >> > spec file): > >> > Rawhide has: 2.0-1.fc32 > >> > F31 has: 1.0-1.fc31 > >> > F30 has: 1.0-1.fc30 > >> > > >> > Next release in Rawhide: 3.0-0.1.20200224git1234abcd > >> > > >> > > >> > Pros: > >> > - Allows triggering two builds from the same commit without manual > >> > intervention > >> > - Emulates what many maintainers do manually today for most use cases > >> > - Can cater for pre-releases (e.g.: by offering different macros or > >> > macro > >> > options for the different use cases) > >> > > >> > Cons: > >> > - Needs the build system for information about builds in this and > >> > other Fedora > >> > versions > >> > - Not easy to reproduce locally because we don’t have > >> > machine-consumable > >> > knowledge about other builds in e.g. the dist-git repo > >> > - Does not allow to generate changelog entries with `version-release` > >> > information for all commits (and this will require a change in our > >> > packaging > >> > guidelines) > >> > > >> > > >> > So these are the results of our current investigations, we are very much > >> > eager > >> > to get your feedback on them and even more eager if you have ideas on > >> > how to > >> > approach/solve some of the challenges mentioned here. > >> > > >> > > >> > Looking forward for the discussion, > >> > > >> > Pierre > >> > For Adam, Nils and myself > >> > >> What is the point of including number of builds into release? I think > >> the Miro's approach solves it. > >> Or is there any other problem except soname bumps? > >> > >> Ad. document - annotated git tags: > >> (-) Editing the changelog would mean allowing to remove the git tags, > >> thus leading to potential issue in build reproducibility > >> > >> That doesn't need to be the case. In rpkg-util, this was resolved by > >> introducing arguments since_tag and until_tag > >> for git_changelog macro > >> (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). > >> That's > >> how it can be prevented for some annotated tag to contribute to changelog. > >> > >> Example: > >> > >> {{{ git_changelog since_tag=name-1.3-1 }}} > >> > >> * Mon Feb 24 2020 clime <cl...@fedoraproject.org> 1.2-1 > >> - manual changelog entry that is used instead of a tag annotation > >> > >> {{{ git_changelog until_tag=name-1.1-1 }}} > >> > >> Removing already pushed git tags is probably not a good idea anyway: > >> https://git-scm.com/docs/git-tag#_on_re_tagging > >> > >> Ad. the following approach for calculating release: > >> - Compute the release field from the number of commits since the last > >> version change: <version>-<commits_number>%{dist} > >> > >> I think it is a good idea. In rpkg-util, it is done similarly, except > >> the release part has more subparts than just > >> two (including %{dist}). > >> > >> The full description is here: > >> https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 > >> but the main difference is that it counts commits from the latest > >> annotated tag which contains (in its name) > >> the current version and the current release number which are extracted > >> from the spec file when > >> creating the tag unless they are specified manually on command line. > >> Commit count is only appended > >> to it if we build from commit which is on top of some annotated tag > >> (i.e. it is itself untagged). > >> > >> Going by just a textual version change in a spec file might be slightly > >> tricky. > >> You would need to go through all the past commits that touched that spec > >> file, > >> keep checking these out and look if the version is changed when compared > >> to the > >> one parsed from the head commit. Possible though. > >> > >> To go back to your original post: > >> > For both the release and the changelog fields we believe using RPM > >> > macros would > >> > satisfy the requirements we have for opting-in/out: > >> > >> By using such RPM macros, you would lose ability to rebuild srpms > >> because it will > >> be only possible to build them when the git context is present but not > >> when they > >> become a standalone thing. I.e. things like this will not work: > >> > >> https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mockbuild/backend.py#L270 > >> > >> That's why I think that such macros should be of a different kind: > >> macros that are computed > >> once when srpm is created and result of which is put _verbatim_ into > >> the spec file so that > >> there is no (re)computation later when srpm is being built and when > >> the git context is already > >> missing. > >> > >> This approach is taken in the rpkg-util project > >> (https://pagure.io/rpkg-util). I really > >> recommend looking at it as I spent more than a year solving this > >> particular problem with > >> changelog and release (but actually not only that problem) and I also > >> optimized the macros there > >> as much as I possibly could. > >> > >> If you want to play around with it, you can download the latest > >> version from here: > >> > >> https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ > >> > >> and try it on this: > >> > >> https://pagure.io/hello_rpkg > >> > >> clime > >> > >> > > >> > > >> > [1]: > >> > https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/ > >> > [2]: https://hackmd.io/kuXOPe74RfepuztBz7pBsg > >> > _______________________________________________ > >> > devel mailing list -- devel@lists.fedoraproject.org > >> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > >> > Fedora Code of Conduct: > >> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> > List Archives: > >> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > >> _______________________________________________ > >> devel mailing list -- devel@lists.fedoraproject.org > >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > > > > _______________________________________________ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
