Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 31. 03. 21 21:52, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RPM-4.17 == Summary == Update RPM to the [https://rpm.org/wiki/Releases/4.17.0 4.17] release. == Owner == * Name: [[User:pmatilai|Panu Matilainen]] * Email: [pmati...@redhat.com] == Detailed Description == RPM 4.17 contains numerous improvements over previous versions * More robust install failure handling * Many macro improvements, in particular much improved Lua integration * Strict checking for unpackaged content in builds * Libraries no longer need executable permission for dependency generation and is automatically removed for non-executable libraries * Long needed transaction APIs enhancements * Improved documentation Panu, in case this was lost in the enthusiastic discussion about the behavior of %exclude, I just wanted to say that I am very exited about all the other new stuff in this release, particularly the Lua stuff. Thanks! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/8/21 10:17 AM, Panu Matilainen wrote: On 4/7/21 4:16 PM, Panu Matilainen wrote: On 4/7/21 2:21 PM, Tomas Orsava wrote: On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. There was good feedback and constructive ideas in the PR. I don't have the energy or the cycles to deal with all this now so I reverted the %exclude-changes upstream, to be revisited at a better time, along with some better options for %check. Thanks everybody for keeping this civil and constructive. Thank you too, Panu, for taking our feedback into account! Best of luck to you and our beloved RPM! Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 4:16 PM, Panu Matilainen wrote: On 4/7/21 2:21 PM, Tomas Orsava wrote: On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. There was good feedback and constructive ideas in the PR. I don't have the energy or the cycles to deal with all this now so I reverted the %exclude-changes upstream, to be revisited at a better time, along with some better options for %check. Thanks everybody for keeping this civil and constructive. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 5:14 PM, Vít Ondruch wrote: Dne 07. 04. 21 v 11:05 Panu Matilainen napsal(a): On 4/7/21 11:45 AM, Panu Matilainen wrote: On 4/6/21 7:40 PM, Vít Ondruch wrote: Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a): On 4/6/21 1:36 PM, Miro Hrončok wrote: For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. This seems like a fairly exotic case to me - okay, a Python-peculiar problem. And a problem of stepping (not saying crossing, because I'm not sure it is) on the boundaries of the %check use-case I suppose. %ghost'ing the files might be one option - I don't know about the Ruby cache case beyond that there is one. There is only `%{gem_cache}` (I assume it was mentioned in the context of `%exclude`, because that has nothing to do with testing). Not shipping this file is enough and we don't ship it just because we don't want to ship file which looks like upstream file but it is not the original upstream file. Moreover we don't really need it for the purposes it is used by upstream, which is restoring the original state of the library. Ah, okay, it's an entirely different case then. Does this file ever get created when running the installed software (by root)? If so, then %ghost'ing it would seem to be the right thing. The chances to have this file created are pretty slim end even if it was created, it would not be trustworthy. If they can get created (even if rare) then you'd want it to be removed along with the package. Which having them %ghost'ed would achieve. If it's just something that needs to be scrubbed on each and every ruby package ever built, we could always add a buildroot policy for it. Just like we could automatically clean .la files rather than manually clean them in thousands of packages... Is it correct assumption that any "random" packages can ship BRP? Interesting idea, I have to give it thought. I guess the concern would be that the packages for the most recent Fedora/RPM won't be compatible with older releases without the BRP. But it should be possible to backport it I guess. Anybody can ship BRP but there's no drop-in mechanism to get then to execute atm, you have to override %__os_install_post for that. So in practise BRPs need to be enabled via central macros in redhat-rpm-config, but there's no reason domain-specific BRPs could not be enabled from there (even if the actual BRP lives in another package). - Panu - Vít ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Panu Matilainen wrote: > I'm starting to think the right thing to do is to move %check to run > after %build rather than %install. Here's one thing that would be affected by such a change: https://docs.fedoraproject.org/en-US/packaging-guidelines/Ada/#_runpaths Our Ada packaging policy has a requirement to run check-rpaths at the end of %check to guard against runpaths slipping into packaged files. It was added on the FPC's request. They wanted it to run as late as possible. Here's the discussion for reference: https://meetbot.fedoraproject.org/teams/fpc/fpc.2013-12-19-17.02.log.html check-rpaths checks files in RPM_BUILD_ROOT, so it would have to be moved to the end of %install if %check would run before %install. I'm aware that there's a proposal to run check-rpaths automatically on all packages: https://pagure.io/packaging-committee/issue/886 I think that's a good idea. If it gets implemented, then we can remove check-rpaths from the Ada spec files – but there might be other similar usecases where something runs in %check to check files in the buildroot, which would break if %check would be moved before %install. Björn Persson pgpfJE5qbRvw7.pgp Description: OpenPGP digital signatur ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 07. 04. 21 v 11:05 Panu Matilainen napsal(a): On 4/7/21 11:45 AM, Panu Matilainen wrote: On 4/6/21 7:40 PM, Vít Ondruch wrote: Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a): On 4/6/21 1:36 PM, Miro Hrončok wrote: For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. This seems like a fairly exotic case to me - okay, a Python-peculiar problem. And a problem of stepping (not saying crossing, because I'm not sure it is) on the boundaries of the %check use-case I suppose. %ghost'ing the files might be one option - I don't know about the Ruby cache case beyond that there is one. There is only `%{gem_cache}` (I assume it was mentioned in the context of `%exclude`, because that has nothing to do with testing). Not shipping this file is enough and we don't ship it just because we don't want to ship file which looks like upstream file but it is not the original upstream file. Moreover we don't really need it for the purposes it is used by upstream, which is restoring the original state of the library. Ah, okay, it's an entirely different case then. Does this file ever get created when running the installed software (by root)? If so, then %ghost'ing it would seem to be the right thing. The chances to have this file created are pretty slim end even if it was created, it would not be trustworthy. If it's just something that needs to be scrubbed on each and every ruby package ever built, we could always add a buildroot policy for it. Just like we could automatically clean .la files rather than manually clean them in thousands of packages... Is it correct assumption that any "random" packages can ship BRP? Interesting idea, I have to give it thought. I guess the concern would be that the packages for the most recent Fedora/RPM won't be compatible with older releases without the BRP. But it should be possible to backport it I guess. Vít OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 2:21 PM, Tomas Orsava wrote: On 4/7/21 12:45 PM, Panu Matilainen wrote: On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. You're right, knowing the scope of the breakage in advance is just not possible. We have similar problems with Python, but given the nature of RPM, you've got it even tougher. I really apologize if it felt like yelling, that was not my intention. No worries. Didn't mean to imply *you* were yelling, this has been entirely civil as far as I'm concerned. I was just referring to years of battling these issues: for every little loophole in rpm we close in order to make it saner we get a backslash, and not always so civilized. It just gets tiresome sometimes. I'm sure you know the feeling. I myself was kicking and screaming through most of the Python 3 transition, so been on the other side of it too :) However, I hope you'll understand why I felt a certain urgency when writing my response. From my own packaging experience the change would break a lot of usage, and other people on this thread only reinforced that this would be a major problem in many areas. And so when I saw that the follow up to this discussion was a PR that would break even more use cases, I grew worried. Yes, message heard. This has been an important discussion to have because it revealed various things I had no idea about, in particular the %exclude linkage to %check. On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. Nod. I'll need to sleep on this all a bit. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 12:45 PM, Panu Matilainen wrote: On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. You're right, knowing the scope of the breakage in advance is just not possible. We have similar problems with Python, but given the nature of RPM, you've got it even tougher. I really apologize if it felt like yelling, that was not my intention. However, I hope you'll understand why I felt a certain urgency when writing my response. From my own packaging experience the change would break a lot of usage, and other people on this thread only reinforced that this would be a major problem in many areas. And so when I saw that the follow up to this discussion was a PR that would break even more use cases, I grew worried. On the other hand, I understand where you're coming from: we have fought battles with unintended use of our tools too (e.g. sudo pip breaking dnf). But given the scope of the breakage here, I would advocate for postponing this change for now. It seems none of us is sure how to square this circle at the moment. All the best, Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 1:12 PM, Tomas Orsava wrote: On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Isn't that exactly what we're doing here? The %check PR may or may not go in, it was linked here so people have a chance to discuss it. Because there's this just discovered linkage between %exclude and %check uses, it probably makes sense for them to go in together. Which could mean doing both, or neither, in 4.17. As in, reverting the %exclude change in 4.17 is entirely possible if it makes sense in the grand scheme things. We cannot possibly know what all the tens of thousands of packages out there are doing, and people will only ever wake up when the change is about to hit them. We've done a dozens of similar changes not because it's fun or because I enjoy getting yelled at for breaking their stuff but because there's no other way to fix ambiguity than making it unambiguous. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 11:38 AM, Panu Matilainen wrote: On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 I understand wanting a perfect system, but I think it's important to realize that rpm is a tool for maintainers to simplify their life and make packaging other software possible. What you're proposing now is to break thousands of packages with no advanced warning, no migration path, just because they're using rpm in a way that wasn't predicted. Please don't do this. If you really want to introduce such backwards incompatible changes, please first work with us maintainers on the migration path and a reasonable timeline to do so. Tomas - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 12:06 PM, Miro Hrončok wrote: On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? Because they kinda go together: the %exclude change revealed that %check is being used in ways it wasn't really intended to be used - just like %exclude was - and both "abuses" come with unwanted side-effects (note quotes, while I know the internal intention, the intended uses were never clearly documented anywhere) Further rationale at https://github.com/rpm-software-management/rpm/pull/1618 - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 07. 04. 21 10:45, Panu Matilainen wrote: I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. This is what I don't understand: The current %exclude change is backwards incompatible and breaks things. How does breaking it even more help this problem? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/7/21 11:45 AM, Panu Matilainen wrote: On 4/6/21 7:40 PM, Vít Ondruch wrote: Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a): On 4/6/21 1:36 PM, Miro Hrončok wrote: For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. This seems like a fairly exotic case to me - okay, a Python-peculiar problem. And a problem of stepping (not saying crossing, because I'm not sure it is) on the boundaries of the %check use-case I suppose. %ghost'ing the files might be one option - I don't know about the Ruby cache case beyond that there is one. There is only `%{gem_cache}` (I assume it was mentioned in the context of `%exclude`, because that has nothing to do with testing). Not shipping this file is enough and we don't ship it just because we don't want to ship file which looks like upstream file but it is not the original upstream file. Moreover we don't really need it for the purposes it is used by upstream, which is restoring the original state of the library. Ah, okay, it's an entirely different case then. Does this file ever get created when running the installed software (by root)? If so, then %ghost'ing it would seem to be the right thing. If it's just something that needs to be scrubbed on each and every ruby package ever built, we could always add a buildroot policy for it. Just like we could automatically clean .la files rather than manually clean them in thousands of packages... But since Ruby was mentioned there, we generally run test suite in `%{_builddir}` (and there are (unfortunately) two possible location, while only one is preferred). Generally, it could be run in `%{buildroot}` with similar results, but it should be discouraged due to possible `%{buildroot}` pollution. Ok, good. Indeed, buildroot should not be used by %check because it can and often does have side-effects. I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. https://github.com/rpm-software-management/rpm/pull/1618 - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 7:40 PM, Vít Ondruch wrote: Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a): On 4/6/21 1:36 PM, Miro Hrončok wrote: For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. This seems like a fairly exotic case to me - okay, a Python-peculiar problem. And a problem of stepping (not saying crossing, because I'm not sure it is) on the boundaries of the %check use-case I suppose. %ghost'ing the files might be one option - I don't know about the Ruby cache case beyond that there is one. There is only `%{gem_cache}` (I assume it was mentioned in the context of `%exclude`, because that has nothing to do with testing). Not shipping this file is enough and we don't ship it just because we don't want to ship file which looks like upstream file but it is not the original upstream file. Moreover we don't really need it for the purposes it is used by upstream, which is restoring the original state of the library. Ah, okay, it's an entirely different case then. Does this file ever get created when running the installed software (by root)? If so, then %ghost'ing it would seem to be the right thing. If it's just something that needs to be scrubbed on each and every ruby package ever built, we could always add a buildroot policy for it. Just like we could automatically clean .la files rather than manually clean them in thousands of packages... But since Ruby was mentioned there, we generally run test suite in `%{_builddir}` (and there are (unfortunately) two possible location, while only one is preferred). Generally, it could be run in `%{buildroot}` with similar results, but it should be discouraged due to possible `%{buildroot}` pollution. Ok, good. Indeed, buildroot should not be used by %check because it can and often does have side-effects. I'm starting to think the right thing to do is to move %check to run after %build rather than %install. That would completely eliminate arguments over what is proper use and should this or that be done etc. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 06. 04. 21 v 16:02 Panu Matilainen napsal(a): On 4/6/21 1:36 PM, Miro Hrončok wrote: For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. This seems like a fairly exotic case to me - okay, a Python-peculiar problem. And a problem of stepping (not saying crossing, because I'm not sure it is) on the boundaries of the %check use-case I suppose. %ghost'ing the files might be one option - I don't know about the Ruby cache case beyond that there is one. There is only `%{gem_cache}` (I assume it was mentioned in the context of `%exclude`, because that has nothing to do with testing). Not shipping this file is enough and we don't ship it just because we don't want to ship file which looks like upstream file but it is not the original upstream file. Moreover we don't really need it for the purposes it is used by upstream, which is restoring the original state of the library. But since Ruby was mentioned there, we generally run test suite in `%{_builddir}` (and there are (unfortunately) two possible location, while only one is preferred). Generally, it could be run in `%{buildroot}` with similar results, but it should be discouraged due to possible `%{buildroot}` pollution. Vít OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 3:01 PM, Tim Landscheidt wrote: Miro Hrončok wrote: […] 2. change %check not to rely on unpackaged files in buildroot That one is non-trivial and depends on the reason it is needed. For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. […] My understanding of the RPM spec sections was always that: - "%prep" is for "./configure", - "%build" is for "make", - "%install" is for "make install", and - "%check" is for "make check". "make check" (usually executed /before/ "make install") works in and on the working directory (https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets). Exactly. Only rpm got this sequence wrong, because %check is executed after %install. If it wasn't for that, we wouldn't be having this conversation at all because then %check would be clearly defined. For the curious, this is how %check came to existence: https://bugzilla.redhat.com/show_bug.cgi?id=64137 I'm kinda wondering here if there wouldn't be any solutions to in this direction - the existing %check is being (ab)used for things its not well suited to because it's positioned in the way it is, and people be better off with something different entirely. To test that the resulting binary package is functional, an- other venue is needed, and in the case of Fedora, for that purpose Fedora CI was created (https://docs.fedoraproject.org/en-US/ci/tests/). That feels like a much cleaner solution than installing some files, testing and then not shipping them because the test environment will be the same as a user's who just installed the package instead of being in the process of building the package. Indeed. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/6/21 1:36 PM, Miro Hrončok wrote: On 06. 04. 21 11:16, Panu Matilainen wrote: On 4/1/21 12:45 AM, Miro Hrončok wrote: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo This still succeeds: %files / %exclude /foo %files foo /foo Many packages do the former in Fedora for various different reasons, namely to, well... exclude files from the package (and not ship them at all). Sometimes a `rm` in %install can be used instead. Sometimes not, because the files are needed in the %{buildroot} for %check but not needed to be shipped. The bottom line is that the buildroot should not contain anything that is not packaged. This has been the basic premise ever since the check for unpackaged files was added somewhere around turn of the millenium, but some loopholes were left around. Yes, %exclude is a loophole. So 'rm -f' at end of %install for undesired material is the expected fix. What %check may or may not do has probably never been designed, much less documented or enforced. It runs after %install yes, so one might assume it's ok/supposed to use what's in buildroot, but then it runs from the build directory, not buildroot. It doesn't matter where does it run *from*, it needs to "see" the files from %{buildroot} because that is what we want to test: The files we ship. The way I see it, %check should be able to use/access buildroot contents, but it certainly should not write there. That might be something to even enforce one day. The packages (that I know about) don't need to actually write there in %check. They just need to to see the files that will be later excluded (e.g. because they belong to a different package that the soon-to-be-built package requires on runtime). And therein lies the problem. Like many things in rpm, %check isn't actually defined in any meaningful way, but in practise it's kinda limited to standalone unit/functional testing and this steps on the border of integration testing. When this change was introduced upstream in November 2020, I've analyzed the impact on Fedora packages. Bare in mind that the data is 4+ months old. https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917 - 1675 packages had %exclude in the spec file - 261 packages FTBFS for unrelated reason (incl. a very limited timeout) - 1414 packages actually tested - 537 packages built successfully, that is ~38% - 877 packages failed with unpackaged files, that is ~62%, list attached When I extrapolate the numbers to compensate the unrelated FTBFS, that's likely more than 1000 affected Fedora packages. OTOH ~500 packages are generated rubygem-* packages, automatically fixable. I'd like to know how are the affected packages supposed to migrate to RPM 4.17 behavior, especially if they cannot remove the files in %install prior to %check. Are they supposed to remove the files at the end of %check instead? What if the package is build without %check? In the order of preference: 1. 'rm -f ' at end of %install That one is simple. I agree. If it is possible, let's do this. We could even automate it if needed. (As a side note I'd say writing such automation is a right thing to do when a change like this is proposed, but I understand that we cannot have everything.) 2. change %check not to rely on unpackaged files in buildroot That one is non-trivial and depends on the reason it is needed. Of course. For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Miro Hrončok wrote: > […] >> 2. change %check not to rely on unpackaged files in buildroot > That one is non-trivial and depends on the reason it is needed. > For example, what is common for Python "namepsace" packages, e.g. > pkg_name.foo. > 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH > to >%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we >(try hard to) exclude $PWD from it. This is crucial to ensure the files >we actually ship are working and the installed file set is complete. >Our macros do this for the packagers. > 2) The %{python3_site...}/pkg_name/ directory and >%{python3_site...}/pkg_name/__init__.py and >%{python3_site...}/pkg_name/__pycache__/ and >%{python3_site...}/pkg_name/__pycache__/__init__...pyc >files must be present in %{buildroot} to successfully run the tests. > 3) The files from (2) must be excluded from the package because *pkg_name* > owns >them, not *pkg_name.foo*. >We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. >The files are not bit-by-bit+metadata identical, >so both packages cannot ship them. > […] My understanding of the RPM spec sections was always that: - "%prep" is for "./configure", - "%build" is for "make", - "%install" is for "make install", and - "%check" is for "make check". "make check" (usually executed /before/ "make install") works in and on the working directory (https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets). To test that the resulting binary package is functional, an- other venue is needed, and in the case of Fedora, for that purpose Fedora CI was created (https://docs.fedoraproject.org/en-US/ci/tests/). That feels like a much cleaner solution than installing some files, testing and then not shipping them because the test environment will be the same as a user's who just installed the package instead of being in the process of building the package. Tim ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 06. 04. 21 11:16, Panu Matilainen wrote: On 4/1/21 12:45 AM, Miro Hrončok wrote: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo This still succeeds: %files / %exclude /foo %files foo /foo Many packages do the former in Fedora for various different reasons, namely to, well... exclude files from the package (and not ship them at all). Sometimes a `rm` in %install can be used instead. Sometimes not, because the files are needed in the %{buildroot} for %check but not needed to be shipped. The bottom line is that the buildroot should not contain anything that is not packaged. This has been the basic premise ever since the check for unpackaged files was added somewhere around turn of the millenium, but some loopholes were left around. Yes, %exclude is a loophole. So 'rm -f' at end of %install for undesired material is the expected fix. What %check may or may not do has probably never been designed, much less documented or enforced. It runs after %install yes, so one might assume it's ok/supposed to use what's in buildroot, but then it runs from the build directory, not buildroot. It doesn't matter where does it run *from*, it needs to "see" the files from %{buildroot} because that is what we want to test: The files we ship. The way I see it, %check should be able to use/access buildroot contents, but it certainly should not write there. That might be something to even enforce one day. The packages (that I know about) don't need to actually write there in %check. They just need to to see the files that will be later excluded (e.g. because they belong to a different package that the soon-to-be-built package requires on runtime). When this change was introduced upstream in November 2020, I've analyzed the impact on Fedora packages. Bare in mind that the data is 4+ months old. https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917 - 1675 packages had %exclude in the spec file - 261 packages FTBFS for unrelated reason (incl. a very limited timeout) - 1414 packages actually tested - 537 packages built successfully, that is ~38% - 877 packages failed with unpackaged files, that is ~62%, list attached When I extrapolate the numbers to compensate the unrelated FTBFS, that's likely more than 1000 affected Fedora packages. OTOH ~500 packages are generated rubygem-* packages, automatically fixable. I'd like to know how are the affected packages supposed to migrate to RPM 4.17 behavior, especially if they cannot remove the files in %install prior to %check. Are they supposed to remove the files at the end of %check instead? What if the package is build without %check? In the order of preference: 1. 'rm -f ' at end of %install That one is simple. I agree. If it is possible, let's do this. We could even automate it if needed. (As a side note I'd say writing such automation is a right thing to do when a change like this is proposed, but I understand that we cannot have everything.) 2. change %check not to rely on unpackaged files in buildroot That one is non-trivial and depends on the reason it is needed. For example, what is common for Python "namepsace" packages, e.g. pkg_name.foo. 1) We want to test installed files, not what is in $PWD, so we set PYTHONPATH to %{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} and we (try hard to) exclude $PWD from it. This is crucial to ensure the files we actually ship are working and the installed file set is complete. Our macros do this for the packagers. 2) The %{python3_site...}/pkg_name/ directory and %{python3_site...}/pkg_name/__init__.py and %{python3_site...}/pkg_name/__pycache__/ and %{python3_site...}/pkg_name/__pycache__/__init__...pyc files must be present in %{buildroot} to successfully run the tests. 3) The files from (2) must be excluded from the package because *pkg_name* owns them, not *pkg_name.foo*. We Require the "toplevel" *pkg_name* package from *pkg_name.foo*. The files are not bit-by-bit+metadata identical, so both packages cannot ship them. Or, let's assume I want to package libfoo-devel for EPEL 9 because RHEL 9 only ships libfoo. And I want to run %check, but only ship the headers. There are plenty situations like this. The case-by-case fix is also impossible to do at scale without a huge heroic effort. 3. short-term, "%_un
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 02. 04. 21 v 12:07 Zbigniew Jędrzejewski-Szmek napsal(a): On Thu, Apr 01, 2021 at 12:33:48AM +0200, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. Thank you for the explanation. I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. That sounds like a plausible concern. But is this something that actually caused real problems, or just a theoretical issue? This is one of the real issues: https://bugzilla.redhat.com/show_bug.cgi?id=878863 Vít For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). If the system was designed like this initially, that'd be fine. But this pattern is widely used and up-to-now there was no indication in the packaging guidelines or rpm output that this is not the recommended pattern. In fact, I know some people preferred the (declarative) %exclude over the (imperative) rm. And Miro raises another issue upthread: there might be packages which require files in %check, and %exclude them. This change would require removing those files at the end of %check, which is rather ugly. Right we have ~1000 packages which will break. There is also an unknown number of non-distro packages which may be affected. I'm not happy about how rpm is changing in a backwards incompatible way. E.g. this means that suddenly ~5% of F33 will not rebuild on F35. I think we should have a strong justification for such a change. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 1:33 AM, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek mailto:zbys...@in.waw.pl>> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > On 31. 03. 21 21:52, Ben Cotton wrote: > >* Strict checking for unpackaged content in builds > > ... > >* Many existing packages will fail to build due to the stricter > >buildroot content checking. Fixing this in the packaging is always > >backwards compatible. We could temporarily set > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > >impact if necessary. > > This is my main concern with this update. > > tl;dr If you %exclude something and there is no other subpackage to > own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. Indeed. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. More generally: what is not there can not cause unwanted side-effects. File-level exclusion is impossible to meaningfully communicate to externally executed scripts, including but not limited to those shipping with rpm itself. There are other benefits further down the road, such as automatic sub-packaging, enforcing %check will not muck with buildroot contents etc. - Panu - -- Kalev ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 12:45 AM, Miro Hrončok wrote: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo This still succeeds: %files / %exclude /foo %files foo /foo Many packages do the former in Fedora for various different reasons, namely to, well... exclude files from the package (and not ship them at all). Sometimes a `rm` in %install can be used instead. Sometimes not, because the files are needed in the %{buildroot} for %check but not needed to be shipped. The bottom line is that the buildroot should not contain anything that is not packaged. This has been the basic premise ever since the check for unpackaged files was added somewhere around turn of the millenium, but some loopholes were left around. Yes, %exclude is a loophole. So 'rm -f' at end of %install for undesired material is the expected fix. What %check may or may not do has probably never been designed, much less documented or enforced. It runs after %install yes, so one might assume it's ok/supposed to use what's in buildroot, but then it runs from the build directory, not buildroot. The way I see it, %check should be able to use/access buildroot contents, but it certainly should not write there. That might be something to even enforce one day. When this change was introduced upstream in November 2020, I've analyzed the impact on Fedora packages. Bare in mind that the data is 4+ months old. https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917 - 1675 packages had %exclude in the spec file - 261 packages FTBFS for unrelated reason (incl. a very limited timeout) - 1414 packages actually tested - 537 packages built successfully, that is ~38% - 877 packages failed with unpackaged files, that is ~62%, list attached When I extrapolate the numbers to compensate the unrelated FTBFS, that's likely more than 1000 affected Fedora packages. OTOH ~500 packages are generated rubygem-* packages, automatically fixable. I'd like to know how are the affected packages supposed to migrate to RPM 4.17 behavior, especially if they cannot remove the files in %install prior to %check. Are they supposed to remove the files at the end of %check instead? What if the package is build without %check? In the order of preference: 1. 'rm -f ' at end of %install 2. change %check not to rely on unpackaged files in buildroot 3. short-term, "%_unpackaged_files_terminate_build 0" in the spec with explanation (similar to eg unsupported architectures) 2. would be case-by-case of course, but an "universal" solution is to create a private install root inside %check, eg --- %check export CHECKROOT=${PWD}/_check %make_install DESTDIR=${CHECKROOT} # ...do what you normally do, just replace BUILDROOT with CHECKROOT --- This way %check will not be messing with the precious to-be-packaged contents. More light-weight options will exist on case-by-case basis, eg typically a cache can be diverted to some other location with environment variables or such. Using `s` in entire rawhide just to compensate this: - is dangerous for other implications of the setting - only postpones the problem to a later time (when we will face the same issue) It's hardly "dangerous", because the only content that will "leak" because of it is buildid links, which is what happens now. It doesn't affect content inclusion, just whether the message is a warning or error. And for a more specific problem, around ~100 Python packages were affected when tested, many of them crucial (e.g. dnf), so this problem will block the upgrade to Python 3.10 if the change lands in Rawhide before we upgrade Python (which is the current plan) until we fix all the affected packages (by at least adding `%global _unpackaged_files_terminate_build 0` to them, which is a tad big hammer, but it will be our last-resort option). Which is why suggested the global "%_unpackaged_files_terminate_build 0" in rawhide: just to move the impact to a less inconvenient time. No kittens will be killed by doing so. - Panu - List of affected Python packages: https://pagure.io/releng/issue/10072#comment-724315 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-o
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Fri, Apr 2, 2021 at 3:34 AM Richard W.M. Jones wrote: > On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > > * Libraries no longer need executable permission for dependency > > generation and is automatically removed for non-executable libraries > > That's interesting - something which affected OCaml packaging: > > https://github.com/ocaml/dune/issues/4148 In a twist of irony, the version of dune that fixes your bug report is now available in Rawhide and F34, so dune no longer removes the executable bits that rpm no longer requires to be present. :-) -- Jerry James http://www.jamezone.org/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 01, 2021 at 12:33:48AM +0200, Kalev Lember wrote: > On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < > zbys...@in.waw.pl> wrote: > > > On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > > > On 31. 03. 21 21:52, Ben Cotton wrote: > > > >* Strict checking for unpackaged content in builds > > > > ... > > > >* Many existing packages will fail to build due to the stricter > > > >buildroot content checking. Fixing this in the packaging is always > > > >backwards compatible. We could temporarily set > > > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > > > >impact if necessary. > > > > > > This is my main concern with this update. > > > > > > tl;dr If you %exclude something and there is no other subpackage to > > > own the files, the build fails: > > > > Whaaat? What is the point of %exclude if not to exclude files from the > > list? Why would rpm upstream want to break this? Seems like a completely > > backwards change that will make packaging harder instead of easier. > > > > %exclude can be used for splitting up packages, so you can do > > %files foo > %exclude bar.so > *.so > > %files bar > bar.so > > > If my understanding is right, the above is what rpm upstream considers > correct use for %exclude. Thank you for the explanation. > I believe the motivation for that change is brp scripts that would still > see the files that are %excluded in files and possibly do wrong things. > Using rm in install doesn't have that problem. That sounds like a plausible concern. But is this something that actually caused real problems, or just a theoretical issue? > For just not packaging some files, rm at the end of %install usually works > just fine (but people have also been using %exclude for that and this > change would break a bunch of packages that do this. I'm unsure if it's a > good thing or not). If the system was designed like this initially, that'd be fine. But this pattern is widely used and up-to-now there was no indication in the packaging guidelines or rpm output that this is not the recommended pattern. In fact, I know some people preferred the (declarative) %exclude over the (imperative) rm. And Miro raises another issue upthread: there might be packages which require files in %check, and %exclude them. This change would require removing those files at the end of %check, which is rather ugly. Right we have ~1000 packages which will break. There is also an unknown number of non-distro packages which may be affected. I'm not happy about how rpm is changing in a backwards incompatible way. E.g. this means that suddenly ~5% of F33 will not rebuild on F35. I think we should have a strong justification for such a change. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 01, 2021 at 11:26:04AM +0200, Miro Hrončok wrote: > On 01. 04. 21 10:47, Vít Ondruch wrote: > > > >Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a): > >>Hello: > >> > >>Miro Hrončok wrote on 2021/04/01 6:45: > >>>On 31. 03. 21 21:52, Ben Cotton wrote: > * Strict checking for unpackaged content in builds > >>> > ... > * Many existing packages will fail to build due to the stricter > buildroot content checking. Fixing this in the packaging is always > backwards compatible. We could temporarily set > `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > impact if necessary. > >>> > >>>This is my main concern with this update. > >>> > >>>tl;dr If you %exclude something and there is no other > >>>subpackage to own the files, the build fails: > >>> > >>> > >>>This fails: > >>> > >>> %install > >>> ... > >>> touch %{buildroot}/foo %{buildroot}/bar > >>> > >>> %files > >>> / > >>> %exclude /foo > >> > >>As the files Miro has attached shows, this affects not a few rubygems > >>related > >>packages. Many rubygems related packages has: %exclude %gem_cache . > > > > > >Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well > >aware of this change and believe me or not, I support the > >intention, mainly because it avoids unintentional side-effects. > > > >However, so far I have not figured alternative (should be probably > >read as elegant) way to do this. Maybe we should generate some > >file lists for the packages and remove the selected files from the > >FS as well as from the file list. Dunno. > > Yeah, I have no problem with "using %exclude like this is wrong and > it was never intended to be abused in this way" but I miss the "this > is how to do it properly" migration guide. Wouldn't the sensible thing be to introduce another keyword to mean that files should not be packaged, eg. %ignore ? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > * Libraries no longer need executable permission for dependency > generation and is automatically removed for non-executable libraries That's interesting - something which affected OCaml packaging: https://github.com/ocaml/dune/issues/4148 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 4/1/21 12:33 AM, Kalev Lember wrote: On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek mailto:zbys...@in.waw.pl>> wrote: On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > On 31. 03. 21 21:52, Ben Cotton wrote: > >* Strict checking for unpackaged content in builds > > ... > >* Many existing packages will fail to build due to the stricter > >buildroot content checking. Fixing this in the packaging is always > >backwards compatible. We could temporarily set > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > >impact if necessary. > > This is my main concern with this update. > > tl;dr If you %exclude something and there is no other subpackage to > own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). This is what I have enforced when reviewing packages, except for Rubygems where the problem is endemic. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 01. 04. 21 10:47, Vít Ondruch wrote: Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a): Hello: Miro Hrončok wrote on 2021/04/01 6:45: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo As the files Miro has attached shows, this affects not a few rubygems related packages. Many rubygems related packages has: %exclude %gem_cache . Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well aware of this change and believe me or not, I support the intention, mainly because it avoids unintentional side-effects. However, so far I have not figured alternative (should be probably read as elegant) way to do this. Maybe we should generate some file lists for the packages and remove the selected files from the FS as well as from the file list. Dunno. Yeah, I have no problem with "using %exclude like this is wrong and it was never intended to be abused in this way" but I miss the "this is how to do it properly" migration guide. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Dne 01. 04. 21 v 0:54 Mamoru TASAKA napsal(a): Hello: Miro Hrončok wrote on 2021/04/01 6:45: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo As the files Miro has attached shows, this affects not a few rubygems related packages. Many rubygems related packages has: %exclude %gem_cache . Just FTR, as a Ruby maintainer and gem2rpm maintainer, I am well aware of this change and believe me or not, I support the intention, mainly because it avoids unintentional side-effects. However, so far I have not figured alternative (should be probably read as elegant) way to do this. Maybe we should generate some file lists for the packages and remove the selected files from the FS as well as from the file list. Dunno. Vít OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Can't wait. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Hello: Miro Hrončok wrote on 2021/04/01 6:45: On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo As the files Miro has attached shows, this affects not a few rubygems related packages. Many rubygems related packages has: %exclude %gem_cache . Regards, Mamoru ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Thu, Apr 1, 2021 at 12:18 AM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: > On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > > On 31. 03. 21 21:52, Ben Cotton wrote: > > >* Strict checking for unpackaged content in builds > > > ... > > >* Many existing packages will fail to build due to the stricter > > >buildroot content checking. Fixing this in the packaging is always > > >backwards compatible. We could temporarily set > > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > > >impact if necessary. > > > > This is my main concern with this update. > > > > tl;dr If you %exclude something and there is no other subpackage to > > own the files, the build fails: > > Whaaat? What is the point of %exclude if not to exclude files from the > list? Why would rpm upstream want to break this? Seems like a completely > backwards change that will make packaging harder instead of easier. > %exclude can be used for splitting up packages, so you can do %files foo %exclude bar.so *.so %files bar bar.so If my understanding is right, the above is what rpm upstream considers correct use for %exclude. For just not packaging some files, rm at the end of %install usually works just fine (but people have also been using %exclude for that and this change would break a bunch of packages that do this. I'm unsure if it's a good thing or not). I believe the motivation for that change is brp scripts that would still see the files that are %excluded in files and possibly do wrong things. Using rm in install doesn't have that problem. -- Kalev ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 11:45:54PM +0200, Miro Hrončok wrote: > On 31. 03. 21 21:52, Ben Cotton wrote: > >* Strict checking for unpackaged content in builds > > ... > >* Many existing packages will fail to build due to the stricter > >buildroot content checking. Fixing this in the packaging is always > >backwards compatible. We could temporarily set > >`%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial > >impact if necessary. > > This is my main concern with this update. > > tl;dr If you %exclude something and there is no other subpackage to > own the files, the build fails: Whaaat? What is the point of %exclude if not to exclude files from the list? Why would rpm upstream want to break this? Seems like a completely backwards change that will make packaging harder instead of easier. Zbyszek > This fails: > > %install > ... > touch %{buildroot}/foo %{buildroot}/bar > > %files > / > %exclude /foo ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On 31. 03. 21 21:52, Ben Cotton wrote: * Strict checking for unpackaged content in builds > ... * Many existing packages will fail to build due to the stricter buildroot content checking. Fixing this in the packaging is always backwards compatible. We could temporarily set `%_unpackaged_files_terminate_build 0` in rawhide to alleviate initial impact if necessary. This is my main concern with this update. tl;dr If you %exclude something and there is no other subpackage to own the files, the build fails: This fails: %install ... touch %{buildroot}/foo %{buildroot}/bar %files / %exclude /foo This still succeeds: %files / %exclude /foo %files foo /foo Many packages do the former in Fedora for various different reasons, namely to, well... exclude files from the package (and not ship them at all). Sometimes a `rm` in %install can be used instead. Sometimes not, because the files are needed in the %{buildroot} for %check but not needed to be shipped. When this change was introduced upstream in November 2020, I've analyzed the impact on Fedora packages. Bare in mind that the data is 4+ months old. https://github.com/rpm-software-management/rpm/pull/1442#issuecomment-731554917 - 1675 packages had %exclude in the spec file - 261 packages FTBFS for unrelated reason (incl. a very limited timeout) - 1414 packages actually tested - 537 packages built successfully, that is ~38% - 877 packages failed with unpackaged files, that is ~62%, list attached When I extrapolate the numbers to compensate the unrelated FTBFS, that's likely more than 1000 affected Fedora packages. OTOH ~500 packages are generated rubygem-* packages, automatically fixable. I'd like to know how are the affected packages supposed to migrate to RPM 4.17 behavior, especially if they cannot remove the files in %install prior to %check. Are they supposed to remove the files at the end of %check instead? What if the package is build without %check? Using `%_unpackaged_files_terminate_build 0` in entire rawhide just to compensate this: - is dangerous for other implications of the setting - only postpones the problem to a later time (when we will face the same issue) And for a more specific problem, around ~100 Python packages were affected when tested, many of them crucial (e.g. dnf), so this problem will block the upgrade to Python 3.10 if the change lands in Rawhide before we upgrade Python (which is the current plan) until we fix all the affected packages (by at least adding `%global _unpackaged_files_terminate_build 0` to them, which is a tad big hammer, but it will be our last-resort option). List of affected Python packages: https://pagure.io/releng/issue/10072#comment-724315 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok afpfs-ng akmods annobin antimicrox antimicroX appstream argyllcms armacycles-ad arpwatch asciidoc audacious-plugin-fc audiocd-kio autoconf awesome aws a52dec bacula bakefile bamf berusky bless blitz blosc bogl booth botan calls cantoolz CCfits CGAL cloog cockatrice colin compizconfig-python conda coturn cowsay cpl crrcsim-addon-models debian-keyring djvulibre dnf docbook5-style-xsl drupal7-active_tags drupal7-boxes drupal7-calendar drupal7-cck drupal7-cs_adaptive_image drupal7-domain_views drupal7-email drupal7-file_entity_inline drupal7-flexifilter drupal7-i18n_boxes drupal7-i18nviews drupal7-languageicons drupal7-language_switcher drupal7-locale_auto_import drupal7-locale_cookie drupal7-l10n_client drupal7-l10n_pconfig drupal7-l10n_server drupal7-stringoverrides drupal7-strongarm drupal7-theme-ninesixty drupal7-translation_helpers drupal7-translation_table drupal7-transliteration drupal7-workbench drupal8 drush dvdbackup erfa etckeeper fbzx fedora-packager fence-agents flare flaw fontopia fwknop gap-pkg-gbnp geany general-purpose-preprocessor ginga glabels glances glpk glusterfs gnome-js-common gnote gpaw gpsim grads gsim85 gtk+extra gutenprint hatari HdrHistogram_c hexter-dssi chipmunk icc-profiles-openicc icu ikiwiki ipython iwd i3 jack-audio-connection-kit kbd knights kqoauth-qt5 libast libcaca libcdaudio libclaw libcsv libdc1394 libdnet libdrm-armada libdsk libdwarf libfc14audiodecoder libfilezilla libgtop2 liblas libmemcached libnss-mysql libnss-pgsql liboping liborigin libpari23 libqmi LibRaw libsigrok libspectrum libstorj liburing libuser libvdpau-va-gl libvorbis libyubikey libzdb lighttpd lxcfs mhash milia mingw-cppunit mingw-dirac mingw-fltk mingw-gsm mingw-python-lxml mingw-python-markupsafe mingw-qt5-qtgraphicaleffects mingw-qt5-qtserialport mISDN moarvm module-build-service mom mon monitor-edid moodle mrtg nbd-runner ncmpc nemiver netbsd-iscsi netmask NetworkManager-strongswan nfs-ganesha nickle nightview numactl numpy ocaml-findlib odcs omniORBpy openal-soft open-vm-tools openvpn openwsman orafce osm-gps-map ots pacemaker pacman pam_mount parallel passenger pcb-rnd pcs pen perl-Algorithm-FastPermute perl-Apache-Session-NoSQL perl-Ca
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
Zbigniew Jędrzejewski-Szmek writes: > On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: >> ** Dynamic spec generation > > Details? My guess would be that this one is meant: https://github.com/rpm-software-management/rpm/pull/1485 signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change proposal: RPM 4.17 (System-Wide Change proposal)
On Wed, Mar 31, 2021 at 03:52:41PM -0400, Ben Cotton wrote: > ** Dynamic spec generation Details? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure