Re: btrfs loses 32-bit application compatibility after a while
* Chris Murphy: > On Thu, Jul 20, 2023, at 11:55 AM, Florian Weimer wrote: >> * Demi Marie Obenour: >> >>> From this thread, it appears that non-LFS 32-bit software is fundamentally >>> unsupportable in the long run, just like software with 32-bit time_t is >>> unsupportable. That leaves two options: >>> >>> 1. Break the ABI, preferably in such a way that causes non-LFS >>>code to fail at load time rather than crashing. >>> >>> 2. Drop 32-bit support from the distribution altogether. >>> >>> It looks like trying to keep 32-bit non-LFS software working will be an >>> endless time sink and is not sustainable in the long term. >> >> My impression is different. It's btrfs that is a poor choice for people >> who want to run 32-bit software, have a lot of file creations/deletions, >> and do not want to reformat and reinstall periodically. The situation >> with XFS, for example, is different because you can supply the inode32 >> mount option and get 32-bit applications going again, maybe after making >> in-place copies of a few files. > > It should be straightforward to have mock create a subvolume for each > chroot instead of a directory. Subvolumes have their own inode pool. Yes, for mock, because of its temporary chroots. I'm more worried about those who encounter the issue with their home directories. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
Dne 21. 07. 23 v 6:32 Chris Murphy napsal(a): It should be straightforward to have mock create a subvolume for each chroot instead of a directory. Subvolumes have their own inode pool. https://github.com/rpm-software-management/mock/issues/1146 PR is welcome. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 20, 2023, at 11:55 AM, Florian Weimer wrote: > * Demi Marie Obenour: > >> From this thread, it appears that non-LFS 32-bit software is fundamentally >> unsupportable in the long run, just like software with 32-bit time_t is >> unsupportable. That leaves two options: >> >> 1. Break the ABI, preferably in such a way that causes non-LFS >>code to fail at load time rather than crashing. >> >> 2. Drop 32-bit support from the distribution altogether. >> >> It looks like trying to keep 32-bit non-LFS software working will be an >> endless time sink and is not sustainable in the long term. > > My impression is different. It's btrfs that is a poor choice for people > who want to run 32-bit software, have a lot of file creations/deletions, > and do not want to reformat and reinstall periodically. The situation > with XFS, for example, is different because you can supply the inode32 > mount option and get 32-bit applications going again, maybe after making > in-place copies of a few files. It should be straightforward to have mock create a subvolume for each chroot instead of a directory. Subvolumes have their own inode pool. Mock already knows how to delete btrfs subvolumes, which is quite a lot more efficient than calling unlinkat() per file and directory. For the many files created and deleted workflow, I think it's worth taking advantage of this. In the meantime, whatever parent directory is used for the chroots could be swapped out with a subvolume for a short term work around. It doesn't sound like the problem happens very quickly. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
* Demi Marie Obenour: > From this thread, it appears that non-LFS 32-bit software is fundamentally > unsupportable in the long run, just like software with 32-bit time_t is > unsupportable. That leaves two options: > > 1. Break the ABI, preferably in such a way that causes non-LFS >code to fail at load time rather than crashing. > > 2. Drop 32-bit support from the distribution altogether. > > It looks like trying to keep 32-bit non-LFS software working will be an > endless time sink and is not sustainable in the long term. My impression is different. It's btrfs that is a poor choice for people who want to run 32-bit software, have a lot of file creations/deletions, and do not want to reformat and reinstall periodically. The situation with XFS, for example, is different because you can supply the inode32 mount option and get 32-bit applications going again, maybe after making in-place copies of a few files. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On 7/20/23 11:06, Florian Weimer wrote: > * Demi Marie Obenour: > >> On 7/17/23 09:51, Florian Weimer wrote: >>> * Daniel P. Berrangé: >>> > But that still raises the question - why does it look like this > started to happen pretty suddenly around June 30? > The list of updates that were applied to builders in that timeframe > doesn't raise any alarm bells (except maybe the 6.3 kernel): > (see https://pagure.io/releng/issue/11531#comment-864471) > I read the release notes for the 6.3 kernel but didn't see any > mentions of BTRFS changes that could explain this. :( Or what 32-bit code is being run in the context of koji that is NOT compiled with LFS ? Is this something silly like configure script tests not enabling LFS, but the resulting applications correctly using LFS ? >>> >>> A dependency used during RPM file creation, libcap, is not fully built >>> in LFS mode. Once we fix that, we know that we'll run into issues with >>> chkconfig and update-alternatives. It's a never-ending source of bugs. >>> It's not a good use of maintainer time. >>> >>> We can't change the overall distribution flags with CFLAGS injection >>> because doing so has ABI implications. >> >> Can they be changed at the next mass rebuild? > > Not if we want to maintain compatibility with non-Fedora software. It > probably doesn't matter for most packages, but doing a per-package > analysis is a lot of work. From this thread, it appears that non-LFS 32-bit software is fundamentally unsupportable in the long run, just like software with 32-bit time_t is unsupportable. That leaves two options: 1. Break the ABI, preferably in such a way that causes non-LFS code to fail at load time rather than crashing. 2. Drop 32-bit support from the distribution altogether. It looks like trying to keep 32-bit non-LFS software working will be an endless time sink and is not sustainable in the long term. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
Thu, Jul 20, 2023 at 5:07 PM Florian Weimer wrote: > > * Demi Marie Obenour: > > > On 7/17/23 09:51, Florian Weimer wrote: > >> * Daniel P. Berrangé: > >> > But that still raises the question - why does it look like this > started to happen pretty suddenly around June 30? > The list of updates that were applied to builders in that timeframe > doesn't raise any alarm bells (except maybe the 6.3 kernel): > (see https://pagure.io/releng/issue/11531#comment-864471) > I read the release notes for the 6.3 kernel but didn't see any > mentions of BTRFS changes that could explain this. :( > >>> > >>> Or what 32-bit code is being run in the context of koji that is > >>> NOT compiled with LFS ? Is this something silly like configure > >>> script tests not enabling LFS, but the resulting applications > >>> correctly using LFS ? > >> > >> A dependency used during RPM file creation, libcap, is not fully built > >> in LFS mode. Once we fix that, we know that we'll run into issues with > >> chkconfig and update-alternatives. It's a never-ending source of bugs. > >> It's not a good use of maintainer time. > >> > >> We can't change the overall distribution flags with CFLAGS injection > >> because doing so has ABI implications. > > > > Can they be changed at the next mass rebuild? > > Not if we want to maintain compatibility with non-Fedora software. It > probably doesn't matter for most packages, but doing a per-package > analysis is a lot of work. While it's not a long-term solution, would it be possible to just "delay things for long enough until we can drop i686 entirely" by modifying specific packages? For example, in this case, libcap was not 64-bit inode aware, and shadow-utils failed to install in the default buildroot (I hope I summarize this right), but modifying libcap to build with support for 64-bit inodes would have solved this, if I understand correctly? -> Pro: Small changes to specific packages -> Con: Things might still fail later in the build process However, since the libcap/shadow-utils thing was the only failure that I saw that affected the default buildroot, at least this would (for now) completely solve the issue (or make much less likely to happen) for several workflows: - buildSRPMfromSCM tasks would no longer randomly fail if assigned to i686 builders - koschei builds would no longer randomly fail (even though they don't even build on i686) - noarch builds could only fail if assigned to i686 (unlucky case) - scratch builds that only include non-i686 architectures would no longer fail - EPEL builds would no longer fail (don't get built for i686) Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
* Demi Marie Obenour: > On 7/17/23 09:51, Florian Weimer wrote: >> * Daniel P. Berrangé: >> But that still raises the question - why does it look like this started to happen pretty suddenly around June 30? The list of updates that were applied to builders in that timeframe doesn't raise any alarm bells (except maybe the 6.3 kernel): (see https://pagure.io/releng/issue/11531#comment-864471) I read the release notes for the 6.3 kernel but didn't see any mentions of BTRFS changes that could explain this. :( >>> >>> Or what 32-bit code is being run in the context of koji that is >>> NOT compiled with LFS ? Is this something silly like configure >>> script tests not enabling LFS, but the resulting applications >>> correctly using LFS ? >> >> A dependency used during RPM file creation, libcap, is not fully built >> in LFS mode. Once we fix that, we know that we'll run into issues with >> chkconfig and update-alternatives. It's a never-ending source of bugs. >> It's not a good use of maintainer time. >> >> We can't change the overall distribution flags with CFLAGS injection >> because doing so has ABI implications. > > Can they be changed at the next mass rebuild? Not if we want to maintain compatibility with non-Fedora software. It probably doesn't matter for most packages, but doing a per-package analysis is a lot of work. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On 7/17/23 09:51, Florian Weimer wrote: > * Daniel P. Berrangé: > >>> But that still raises the question - why does it look like this >>> started to happen pretty suddenly around June 30? >>> The list of updates that were applied to builders in that timeframe >>> doesn't raise any alarm bells (except maybe the 6.3 kernel): >>> (see https://pagure.io/releng/issue/11531#comment-864471) >>> I read the release notes for the 6.3 kernel but didn't see any >>> mentions of BTRFS changes that could explain this. :( >> >> Or what 32-bit code is being run in the context of koji that is >> NOT compiled with LFS ? Is this something silly like configure >> script tests not enabling LFS, but the resulting applications >> correctly using LFS ? > > A dependency used during RPM file creation, libcap, is not fully built > in LFS mode. Once we fix that, we know that we'll run into issues with > chkconfig and update-alternatives. It's a never-ending source of bugs. > It's not a good use of maintainer time. > > We can't change the overall distribution flags with CFLAGS injection > because doing so has ABI implications. Can they be changed at the next mass rebuild? -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
* Daniel P. Berrangé: >> But that still raises the question - why does it look like this >> started to happen pretty suddenly around June 30? >> The list of updates that were applied to builders in that timeframe >> doesn't raise any alarm bells (except maybe the 6.3 kernel): >> (see https://pagure.io/releng/issue/11531#comment-864471) >> I read the release notes for the 6.3 kernel but didn't see any >> mentions of BTRFS changes that could explain this. :( > > Or what 32-bit code is being run in the context of koji that is > NOT compiled with LFS ? Is this something silly like configure > script tests not enabling LFS, but the resulting applications > correctly using LFS ? A dependency used during RPM file creation, libcap, is not fully built in LFS mode. Once we fix that, we know that we'll run into issues with chkconfig and update-alternatives. It's a never-ending source of bugs. It's not a good use of maintainer time. We can't change the overall distribution flags with CFLAGS injection because doing so has ABI implications. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 11:47:39PM +0200, Fabio Valentini wrote: > On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen wrote: > > > > On 7/14/23 6:53 AM, Florian Weimer wrote: > > > * Neal Gompa: > > > > > >> On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini > > >> wrote: > > >>> > > >>> On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer > > >>> wrote: > > > > Fedora lists are hostile to upstream collaboration via cross-posting, > > so > > I can only forward this for your information. > > > > This causes problems with the i686 builders. > > >>> > > >>> I wonder how this only started to happen recently? Has something > > >>> changed in BTRFS with the 6.3 kernel? > > >>> This only started happening a few days after builders were rebooted at > > >>> the end of June to apply updates (and kernel 6.3 was among those > > >>> updates, as far as I can tell). > > >>> > > >> > > >> This was always possible. I'm curious as to why it took so long for us > > >> to hit it, though. > > >> > > >> The recommended solution is to create a new subvolume for these > > >> environments, since the inode count is reset for each subvolume. > > > > > > What about impact beyond the builders? > > > > > > Are end users are expected to do this? Do we have a tool for this? > > > > FWIW, 64 bit inodes have existed in some Linux filesystems for a very > > long time. On XFS, you'll get them by default - and quickly, not after > > extended use - on a filesystem of sufficient size (around 1T-2T by > > default, if I remember right.) > > > > XFS does have a hack^Wmount option to force inodes into the 32-bit > > range, but just FWIW we almost never see users running into problems > > with 32-bit applications (but maybe because they know about the mount > > option...) > > But that still raises the question - why does it look like this > started to happen pretty suddenly around June 30? > The list of updates that were applied to builders in that timeframe > doesn't raise any alarm bells (except maybe the 6.3 kernel): > (see https://pagure.io/releng/issue/11531#comment-864471) > I read the release notes for the 6.3 kernel but didn't see any > mentions of BTRFS changes that could explain this. :( Or what 32-bit code is being run in the context of koji that is NOT compiled with LFS ? Is this something silly like configure script tests not enabling LFS, but the resulting applications correctly using LFS ? With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 10:46:40PM -, Scott Talbert wrote: > > On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen > wrote: > > > > But that still raises the question - why does it look like this > > started to happen pretty suddenly around June 30? > > The list of updates that were applied to builders in that timeframe > > doesn't raise any alarm bells (except maybe the 6.3 kernel): > > (see https://pagure.io/releng/issue/11531#comment-864471) > > I read the release notes for the 6.3 kernel but didn't see any > > mentions of BTRFS changes that could explain this. :( > > When did the builders get switched to btrfs? May of last year. > Could it just be a matter of accumulating enough inodes since then? Well, not completely. As I noted earlier, usually I do a full re-install of builders after each release. For f38 I didn't... I just updated them from 37->38. So, they were around since f37, but normally would have been re-installed then. Also, buildhw-x86 instances don't seem affected yet, but I think looking more closely at it it's because we have the weight lower on those in koji. They only have a weight of 2, the buildvm-x86 builders have a weight of 4, so they would roughly get 2x the number of builds. Anyhow, please do let me know if you see this now... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
> On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen wrote: > > But that still raises the question - why does it look like this > started to happen pretty suddenly around June 30? > The list of updates that were applied to builders in that timeframe > doesn't raise any alarm bells (except maybe the 6.3 kernel): > (see https://pagure.io/releng/issue/11531#comment-864471) > I read the release notes for the 6.3 kernel but didn't see any > mentions of BTRFS changes that could explain this. :( When did the builders get switched to btrfs? Could it just be a matter of accumulating enough inodes since then? Scott ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On 7/14/23 4:47 PM, Fabio Valentini wrote: On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen wrote: On 7/14/23 6:53 AM, Florian Weimer wrote: * Neal Gompa: On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini wrote: On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: Fedora lists are hostile to upstream collaboration via cross-posting, so I can only forward this for your information. This causes problems with the i686 builders. I wonder how this only started to happen recently? Has something changed in BTRFS with the 6.3 kernel? This only started happening a few days after builders were rebooted at the end of June to apply updates (and kernel 6.3 was among those updates, as far as I can tell). This was always possible. I'm curious as to why it took so long for us to hit it, though. The recommended solution is to create a new subvolume for these environments, since the inode count is reset for each subvolume. What about impact beyond the builders? Are end users are expected to do this? Do we have a tool for this? FWIW, 64 bit inodes have existed in some Linux filesystems for a very long time. On XFS, you'll get them by default - and quickly, not after extended use - on a filesystem of sufficient size (around 1T-2T by default, if I remember right.) XFS does have a hack^Wmount option to force inodes into the 32-bit range, but just FWIW we almost never see users running into problems with 32-bit applications (but maybe because they know about the mount option...) But that still raises the question - why does it look like this started to happen pretty suddenly around June 30? The list of updates that were applied to builders in that timeframe doesn't raise any alarm bells (except maybe the 6.3 kernel): (see https://pagure.io/releng/issue/11531#comment-864471) I read the release notes for the 6.3 kernel but didn't see any mentions of BTRFS changes that could explain this. :( Sure - I can't speak to what might have changed in btrfs, sorry - just saying that binaries that choke on 64 bit inodes are going to get less and less usable over time, I think. One thing you could consider if you're ever really in a pinch is an ld_preload that fakes up a 32-bit inode number for i.e. stat. The vast majority of applications don't care about the inode number at all, and it's really kind of unfortunate to fail the whole stat call just for that. -Eric ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen wrote: > > On 7/14/23 6:53 AM, Florian Weimer wrote: > > * Neal Gompa: > > > >> On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini > >> wrote: > >>> > >>> On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer > >>> wrote: > > Fedora lists are hostile to upstream collaboration via cross-posting, so > I can only forward this for your information. > > This causes problems with the i686 builders. > >>> > >>> I wonder how this only started to happen recently? Has something > >>> changed in BTRFS with the 6.3 kernel? > >>> This only started happening a few days after builders were rebooted at > >>> the end of June to apply updates (and kernel 6.3 was among those > >>> updates, as far as I can tell). > >>> > >> > >> This was always possible. I'm curious as to why it took so long for us > >> to hit it, though. > >> > >> The recommended solution is to create a new subvolume for these > >> environments, since the inode count is reset for each subvolume. > > > > What about impact beyond the builders? > > > > Are end users are expected to do this? Do we have a tool for this? > > FWIW, 64 bit inodes have existed in some Linux filesystems for a very > long time. On XFS, you'll get them by default - and quickly, not after > extended use - on a filesystem of sufficient size (around 1T-2T by > default, if I remember right.) > > XFS does have a hack^Wmount option to force inodes into the 32-bit > range, but just FWIW we almost never see users running into problems > with 32-bit applications (but maybe because they know about the mount > option...) But that still raises the question - why does it look like this started to happen pretty suddenly around June 30? The list of updates that were applied to builders in that timeframe doesn't raise any alarm bells (except maybe the 6.3 kernel): (see https://pagure.io/releng/issue/11531#comment-864471) I read the release notes for the 6.3 kernel but didn't see any mentions of BTRFS changes that could explain this. :( Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On 7/14/23 17:32, Chris Adams wrote: > Once upon a time, Eric Sandeen said: >> XFS does have a hack^Wmount option to force inodes into the 32-bit >> range, but just FWIW we almost never see users running into problems >> with 32-bit applications (but maybe because they know about the >> mount option...) > > Aren't even most 32-bit applications compiled with LFS? It looks like > anything with _GNU_SOURCE will get it, and IIRC that's not uncommon. > I'm curious how many 32-bit binaries are looking at inode numbers and > are NOT already LFS compatible. I recommending that the entire distro be compiled with LFS. The non-LFS ABI is obsolete for exactly this reason. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
Once upon a time, Eric Sandeen said: > XFS does have a hack^Wmount option to force inodes into the 32-bit > range, but just FWIW we almost never see users running into problems > with 32-bit applications (but maybe because they know about the > mount option...) Aren't even most 32-bit applications compiled with LFS? It looks like anything with _GNU_SOURCE will get it, and IIRC that's not uncommon. I'm curious how many 32-bit binaries are looking at inode numbers and are NOT already LFS compatible. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On 7/14/23 6:53 AM, Florian Weimer wrote: * Neal Gompa: On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini wrote: On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: Fedora lists are hostile to upstream collaboration via cross-posting, so I can only forward this for your information. This causes problems with the i686 builders. I wonder how this only started to happen recently? Has something changed in BTRFS with the 6.3 kernel? This only started happening a few days after builders were rebooted at the end of June to apply updates (and kernel 6.3 was among those updates, as far as I can tell). This was always possible. I'm curious as to why it took so long for us to hit it, though. The recommended solution is to create a new subvolume for these environments, since the inode count is reset for each subvolume. What about impact beyond the builders? Are end users are expected to do this? Do we have a tool for this? FWIW, 64 bit inodes have existed in some Linux filesystems for a very long time. On XFS, you'll get them by default - and quickly, not after extended use - on a filesystem of sufficient size (around 1T-2T by default, if I remember right.) XFS does have a hack^Wmount option to force inodes into the 32-bit range, but just FWIW we almost never see users running into problems with 32-bit applications (but maybe because they know about the mount option...) -Eric ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 09:28:15AM -0700, Kevin Fenzi wrote: ...snip... > > * Today / soon, I will just reinstall all the buildvm-x86 vm's. > Thats pretty trivial to do (thanks ansible). It may cause some builds to > restart as I move things around, but it shouldn't be too much. ok, this is done. All the buildvm-x86* build vm's (all 32 of them) are reinstalled. Please let me know if you see this i686 build failure, and what exact machine you see it on. Thanks, kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 10:28:36AM -0400, Neal Gompa wrote: > > > > The 32-bit binary is what suffers the consequences. > > > > The workload that /causes/ the problem can be anything which creates > > lots of files. > > > > I am aware of that, but having something concretely easily > reproducible that appears realistic makes it a lot easier for me to > talk to them about doing something to address it. Otherwise it's all > just hypothetical and the conversation gets difficult. A busy mail server using postfix and creating / deleting tons of queue files as it processes? I guess it's really hard to come up with another case without specific usage patterns (ie, deleting a ton of files and recreating them over and over, some websites might do this?) Anyhow, How about this: * Today / soon, I will just reinstall all the buildvm-x86 vm's. Thats pretty trivial to do (thanks ansible). It may cause some builds to restart as I move things around, but it shouldn't be too much. * If the problem seems to go away, then great and we can work on a longer term solution, which could be some btrfs changes, switching filesystems, dropping i686, etc. ;) * If the problems persist, I can just reinstall again with xfs and the inode32 mount flag. * If that doesn't fix it, we may need to delay the mass rebuild and sort out another fix. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 9:58 AM Daniel P. Berrangé wrote: > > On Fri, Jul 14, 2023 at 09:49:04AM -0400, Neal Gompa wrote: > > On Fri, Jul 14, 2023 at 8:48 AM Florian Weimer wrote: > > > > > > * Miroslav Suchý: > > > > > > > Dne 14. 07. 23 v 13:53 Florian Weimer napsal(a): > > > >> What about impact beyond the builders? > > > >> > > > >> Are end users are expected to do this? Do we have a tool for this? > > > > > > > > Close to zero. You have to do a really lots of builds. And keep the > > > > buildroots of these (failed) builds. > > > > > > This is not restricted to mock or Koji, though. There just needs to be > > > one application that creates and deletes many files (or produces other > > > events that causes object ID allocation in btrfs). > > > > > > For me it's going to be slow enough that the file system will likely be > > > recreated before I encounter the 31-bit object ID limit. But I wonder > > > if that's true for everyone. > > > > > > > I don't know of a 32-bit workload that would trigger this reliably and > > reasonably quickly, but if there is one, then I would be interested in > > knowing about it. > > The 32-bit binary is what suffers the consequences. > > The workload that /causes/ the problem can be anything which creates > lots of files. > I am aware of that, but having something concretely easily reproducible that appears realistic makes it a lot easier for me to talk to them about doing something to address it. Otherwise it's all just hypothetical and the conversation gets difficult. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 09:49:04AM -0400, Neal Gompa wrote: > On Fri, Jul 14, 2023 at 8:48 AM Florian Weimer wrote: > > > > * Miroslav Suchý: > > > > > Dne 14. 07. 23 v 13:53 Florian Weimer napsal(a): > > >> What about impact beyond the builders? > > >> > > >> Are end users are expected to do this? Do we have a tool for this? > > > > > > Close to zero. You have to do a really lots of builds. And keep the > > > buildroots of these (failed) builds. > > > > This is not restricted to mock or Koji, though. There just needs to be > > one application that creates and deletes many files (or produces other > > events that causes object ID allocation in btrfs). > > > > For me it's going to be slow enough that the file system will likely be > > recreated before I encounter the 31-bit object ID limit. But I wonder > > if that's true for everyone. > > > > I don't know of a 32-bit workload that would trigger this reliably and > reasonably quickly, but if there is one, then I would be interested in > knowing about it. The 32-bit binary is what suffers the consequences. The workload that /causes/ the problem can be anything which creates lots of files. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Fri, Jul 14, 2023 at 8:48 AM Florian Weimer wrote: > > * Miroslav Suchý: > > > Dne 14. 07. 23 v 13:53 Florian Weimer napsal(a): > >> What about impact beyond the builders? > >> > >> Are end users are expected to do this? Do we have a tool for this? > > > > Close to zero. You have to do a really lots of builds. And keep the > > buildroots of these (failed) builds. > > This is not restricted to mock or Koji, though. There just needs to be > one application that creates and deletes many files (or produces other > events that causes object ID allocation in btrfs). > > For me it's going to be slow enough that the file system will likely be > recreated before I encounter the 31-bit object ID limit. But I wonder > if that's true for everyone. > I don't know of a 32-bit workload that would trigger this reliably and reasonably quickly, but if there is one, then I would be interested in knowing about it. We used to have something in Btrfs to mitigate this called inode_cache, but it was turned into a no-op in 5.11 and the documentation was removed in 5.12: https://github.com/kdave/btrfs-progs/commit/1af37385e258aabc684484aff8e5def442ada9bb If we have a workload example that subvolumes don't work for, then I'm happy to advocate upstream to do something to address the problem more directly. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
* Miroslav Suchý: > Dne 14. 07. 23 v 13:53 Florian Weimer napsal(a): >> What about impact beyond the builders? >> >> Are end users are expected to do this? Do we have a tool for this? > > Close to zero. You have to do a really lots of builds. And keep the > buildroots of these (failed) builds. This is not restricted to mock or Koji, though. There just needs to be one application that creates and deletes many files (or produces other events that causes object ID allocation in btrfs). For me it's going to be slow enough that the file system will likely be recreated before I encounter the 31-bit object ID limit. But I wonder if that's true for everyone. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
Dne 14. 07. 23 v 13:53 Florian Weimer napsal(a): What about impact beyond the builders? Are end users are expected to do this? Do we have a tool for this? Close to zero. You have to do a really lots of builds. And keep the buildroots of these (failed) builds. For normal users the buildroots are reused. No chance to have so many inodes. Copr terminate builders and spin up new one. And when build finish Copr retrieve only interresting artifacts. So it is no issue there. So it is only Koji. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
* Neal Gompa: > On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini wrote: >> >> On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: >> > >> > Fedora lists are hostile to upstream collaboration via cross-posting, so >> > I can only forward this for your information. >> > >> > This causes problems with the i686 builders. >> >> I wonder how this only started to happen recently? Has something >> changed in BTRFS with the 6.3 kernel? >> This only started happening a few days after builders were rebooted at >> the end of June to apply updates (and kernel 6.3 was among those >> updates, as far as I can tell). >> > > This was always possible. I'm curious as to why it took so long for us > to hit it, though. > > The recommended solution is to create a new subvolume for these > environments, since the inode count is reset for each subvolume. What about impact beyond the builders? Are end users are expected to do this? Do we have a tool for this? Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 13, 2023 at 02:28:55PM +0200, Fabio Valentini wrote: > On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: > > > > Fedora lists are hostile to upstream collaboration via cross-posting, so > > I can only forward this for your information. > > > > This causes problems with the i686 builders. > > I wonder how this only started to happen recently? Has something > changed in BTRFS with the 6.3 kernel? > This only started happening a few days after builders were rebooted at > the end of June to apply updates (and kernel 6.3 was among those > updates, as far as I can tell). Could be kernel related. Or it could be caused by the fact that I just upgraded builders from f37 to f38 this time instead of a clean reinstall. So, perhaps since they had been around longer the fs issue was reached more quickly? But... I haven't reinstalled the buildhw-x86 boxes in a while either, so I would expect we would have hit it on them before now if that was the case. :( kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 13, 2023 at 9:15 AM Fabio Valentini wrote: > > On Thu, Jul 13, 2023 at 2:53 PM Neal Gompa wrote: > > > > On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini > > wrote: > > > > > > On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer > > > wrote: > > > > > > > > Fedora lists are hostile to upstream collaboration via cross-posting, so > > > > I can only forward this for your information. > > > > > > > > This causes problems with the i686 builders. > > > > > > I wonder how this only started to happen recently? Has something > > > changed in BTRFS with the 6.3 kernel? > > > This only started happening a few days after builders were rebooted at > > > the end of June to apply updates (and kernel 6.3 was among those > > > updates, as far as I can tell). > > > > > > > This was always possible. I'm curious as to why it took so long for us > > to hit it, though. > > > > The recommended solution is to create a new subvolume for these > > environments, since the inode count is reset for each subvolume. > > > > This might require some work in Mock to make it do that instead of > > creating a directory, but it should be fairly straightforward. > > I'm quite certain mock used to create btrfs subvolumes for at least > *some* purposes (because I used to need to clean them up manually > sometimes), but doesn't seem to do it anymore. > That might be what is causing the problems now. Because creating subvolumes for each chroot avoids this issue. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 13, 2023 at 2:53 PM Neal Gompa wrote: > > On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini wrote: > > > > On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: > > > > > > Fedora lists are hostile to upstream collaboration via cross-posting, so > > > I can only forward this for your information. > > > > > > This causes problems with the i686 builders. > > > > I wonder how this only started to happen recently? Has something > > changed in BTRFS with the 6.3 kernel? > > This only started happening a few days after builders were rebooted at > > the end of June to apply updates (and kernel 6.3 was among those > > updates, as far as I can tell). > > > > This was always possible. I'm curious as to why it took so long for us > to hit it, though. > > The recommended solution is to create a new subvolume for these > environments, since the inode count is reset for each subvolume. > > This might require some work in Mock to make it do that instead of > creating a directory, but it should be fairly straightforward. I'm quite certain mock used to create btrfs subvolumes for at least *some* purposes (because I used to need to clean them up manually sometimes), but doesn't seem to do it anymore. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini wrote: > > On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: > > > > Fedora lists are hostile to upstream collaboration via cross-posting, so > > I can only forward this for your information. > > > > This causes problems with the i686 builders. > > I wonder how this only started to happen recently? Has something > changed in BTRFS with the 6.3 kernel? > This only started happening a few days after builders were rebooted at > the end of June to apply updates (and kernel 6.3 was among those > updates, as far as I can tell). > This was always possible. I'm curious as to why it took so long for us to hit it, though. The recommended solution is to create a new subvolume for these environments, since the inode count is reset for each subvolume. This might require some work in Mock to make it do that instead of creating a directory, but it should be fairly straightforward. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: btrfs loses 32-bit application compatibility after a while
On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer wrote: > > Fedora lists are hostile to upstream collaboration via cross-posting, so > I can only forward this for your information. > > This causes problems with the i686 builders. I wonder how this only started to happen recently? Has something changed in BTRFS with the 6.3 kernel? This only started happening a few days after builders were rebooted at the end of June to apply updates (and kernel 6.3 was among those updates, as far as I can tell). Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
