Re: Discussion: what would not blocking on btrfs look like?
The language being used by Anaconda team suggests the "Btrfs is not a priority" and should be unsupported, is a decision that has already happened. This discussion, in this thread, is about how to handle that decision in UI/Ux. When and where did this decision get made? How do outside contributors get the information they need to know their efforts are worthwhile? I have hundreds of hours invested in Anaconda testing, perhaps 1/2 not related to Btrfs, over ~8 years. I would like answers to these questions. -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Mon, Sep 09, 2019 at 01:43:00PM -0400, David Lehman wrote: > On Thu, 2019-09-05 at 18:01 -0700, Adam Williamson wrote: > > On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote: > > > On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote: > > > > Hey folks! > > > > > > Hi Adam! Thanks for bringing this up again. > > > > > > > So...what should we do? Here are the options as I see 'em: > > > > > > > > 1. Keep supporting btrfs > > > > 2. Just modify the criterion with a btrfs exception, even if it's > > > > weird > > > > 3. Rewrite the criterion entirely > > > > 4. Keep btrfs support in the installer (and blivet-gui) but hide > > > > it > > > > as > > > > we used to - require a special boot argument for it to be visible > > > > 5. Drop btrfs support from the installer > > > > > > I like option 3 most. The current criteria have always seemed, to > > > me, > > > too vague. I'd be happy to help hash out the details if/when it > > > happens. > > > > Thanks for the offer. > > > > So aside from the 'fun' of drafting very specific rules, my concern > > with #3 is we would then potentially be shipping an installer that > > presents things as roughly equal choices which are not in fact > > equally > > supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one > > of those we commit to making sure is working, one of them we don't. > > > That to me is concerning; in this scenario I'd prefer we indicate > > somehow, somewhere, that all the choices are not equally guaranteed > > to > > be reliable. WDYT? > > > Two off-the-cuff ideas: > > 1. every fs (and device?) type in the combo/dropdown has "(Supported)" > or "(Unsupported)" postfix, eg: "xfs (Supported)" or "BTRFS > (Unsupported)" > 2. every unsupported fs type has a corresponding kernel arg, eg: > "inst.btrfs" or "inst.fs.btrfs" which is required to get that > unsupported fs type in the GUI list > > I could live with either, personally. > Why bother hiding it at all? I get that there's no btrfs expertise at Red Hat, but there's not any reason to indicate that it's bad for the user to choose it. Suse ships it, all of Facebook is on it, it's not like we're talking about hfs+ here or anything. I get the discussions around wether it's a release blocker, I don't really understand why the installer needs to be changed? Thanks, Josef ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Thu, Aug 29, 2019 at 4:23 AM wrote: > > On Tue, 2019-08-27 at 14:59 -0700, Adam Williamson wrote: > > On Tue, 2019-08-27 at 13:25 -0400, David Cantrell wrote: > > > > Josh, to be fair, I can see Neal's point here. In a way you seem > > > > to be > > > > kinda sending him in circles here: "anaconda team doesn't have > > > > the > > > > time/resources to invest in btrfs development, but you can help > > > > by > > > > sending them pull requests. Oh, you sent them a pull request and > > > > they > > > > rejected it? Well, it's because they don't have time/resources > > > > for > > > > btrfs development..." You're right that one rejected PR doesn't > > > > necessarily indicate a contribution model problem, but putting > > > > the > > > > wider issue aside, a very simple one-liner with a major impact on > > > > btrfs > > > > functionality being rejected *does* seem to say a lot about the > > > > prospects of any btrfs-related work being accepted. > > > > > > > > Putting myself in Neal's shoes, given my experience with that PR > > > > and > > > > other attempts to help out with btrfs in anaconda, why would I > > > > invest > > > > my time and effort to write another one when it seems extremely > > > > likely > > > > it would be rejected? > > > > > > There's an assumption here that when someone is asked to send a PR, > > > it > > > will always be accepted. > > > > No, that's not what I'm assuming, but if we (Red Hat) tell people > > that > > it would be a good idea to send PRs, we should at least be > > *potentially* willing to accept them. We should not be saying 'send > > PRs' if there is no chance of them being accepted. And it would be > > nicest if we gave people a roadmap: here are the specific things you > > can do that would be acceptable to us as a way to continue including > > btrfs handling in the installer. > > Sorry but I have to react on this. It's killing me how many of you > people here are telling Anaconda does not accept patches. That is > really not true. We have smaller amount of contributions from community > that is partially our fault because of a lack of documentation but > mainly it's really hard to develop & test Anaconda and most users see > it just once in a few years so it's not bothering them so much. I guess > most of the installers are in the same situation. > I sat on this for a few days, as I wanted to cool down and think about how to reply to this. As of this moment, I have directly and indirectly contributed to both the Red Hat and SUSE installers, and yes it's definitely true that both installers have some of the worst interaction models for contributors I've ever seen. YaST's contribution model and process seems to be somewhat better, because their team has components being used by other people. Their libyui components are used by Fedora and Mageia for dnfdragora, and the rest of ManaTools in Mageia uses it too. The YaST control center is a cornerstone in the user experience in SUSE distributions, so there are contributors from their community who do work on the main YaST components. The Calamares installer stack has a pretty healthy community, but our community has an interesting aversion to all things Qt/KDE even though the installer works fine on Fedora... But the point I'm trying to make is there is no reason for Anaconda to be so difficult. Unfortunately, it is. > Please before any of you will tell again that Anaconda team is not > accepting patches, please look on the last few years history. How many > patches were "rejected" and how many were accepted. There are just a > few patches which weren't accepted and basically only a few PR (I would > even say the only one) for BTRFS. That is unfortunate but it's not all > our contributions. > I also took the opportunity look back at over the last 400 merged pull requests by paging through GitHub. Now I don't exactly have firm numbers, but in the past 400 pull requests, I saw less than a dozen pull requests merged from non-RHers. Half of them were from Pat from the Scientific Linux team. One of them was a documentation fix from some person I don't recognize with no clear affiliation. If I page back *slightly more* then the next PR from a non Red Hatter that was merged was *mine* fixing Anaconda's package exclusion feature for kickstarts (which was nearly two years old when it was merged). Of the 42 currently open pull requests, 4 are from people I could clearly identify as not from Red Hatters. Of the ones that are from Red Hatters, 7 appear to not be from members of the Red Hat Installer team or the Red Hat Bootloader team as I know of them. The lag time to response *is* improving. It's gone down from years to months, with the most recent pull request getting a comment within a week, and then stalling out after that. So it may even improve to weeks! As Adam pointed out, there's literally no reason for me to attempt more sophisticated changes to Anaconda if a simple one-liner can't get merged. And I didn't even ma
Re: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 1:07 PM Josef Bacik wrote: > > On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote: > > Fedora chugs along at the rate of daily upstream Linus snapshots. If > > you're hitting and fixing issues before Fedora users see them, I'm > > curious why Fedora users would ever see them. > > > > Where does the lag come from? Are the fixes queued internally? > > Staged in an upstream subsystem tree? Is there a way for interested > > btrfs people to proactively just get those fixed in Fedora before > > users hit them? > > For this particular example we saw the problem in testing and had a patch on > the > mailinglist before you hit the problem. It was in a tree and sent to Linus, > and > was merged the day after the bugzilla was reported. So yes before users see > them, unless they are subscribed to the daily snapshots, which I assume is > just > for testing right? Or were you guys going to ship 5.3-rc0? > > On one hand I understand all of the consternation around making btrfs bugs > blockers for Fedora, but on the other hand it seems a bit silly to be having > this conversation at all based on hitting a bug that went into the merge > window > and then was fixed before rc1 was even cut. Thanks, It is silly, if it's really safe to say that Btrfs won't be the default file system in any release blocking deliverable. Having blocker status was always a means to that end. But right now it's maybe three (?) automated tests that depend on Btrfs working. If Fedora Workstation defaulted to Btrfs, that's dozens or even hundreds of tests? Adam? Bug fix was merged in rc2. The patch on linux-btrfs@ 25 Jul 2019 11:27:29 +0300 which is just before kernel-5.3.0-0.rc1.git3.1.fc31, 2019-07-25 21:01:20 Plausibly it was in all rc0 and rc1 kernels. What if this deadlock were happening in ext4 for all rc0 and rc1 kernels? What questions get asked? How did the bug not get caught by xfstests before it got into linux-next? Does anyone pop a gasket on lkml? Is this just a weird sometimes it happens rarely kind of thing? I don't know the exact nature of the bug, which goes to the kernel team's point that someone who does know needs to tell them the autopsy summary so they can really assess the default fs question. And another question for QA. If it were Btrfs by default for Workstation, would you just convert all the tests that rely only on ext4 now to Btrfs? Or duplicate those tests so you can run them in parallel? How much more testing is that and what's the impact on time and resources? -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 1:30 PM Laura Abbott wrote: > > > > I also think there are other perspectives that might at least > > potentially be useful here. Right now we've mainly heard from a couple > > of community folks who are very passionate about btrfs, and Red Hat > > folks from anaconda/kernel development and RHEL management who > > essentially see it as a burden that is not aligned with their > > priorities. Is that all the perspectives we have to make a decision > > with? I think we should at least talk to the Facebook team that > > apparently uses btrfs on Fedora extensively about whether they'd be sad > > to see installer btrfs support die and, if so, whether they'd perhaps > > be interested in helping support it. And more broadly, community folks > > on all the lists this is going to: are there more people who actually > > are interested in this functionality and would be sad to see it go? If > > btrfs isn't a part of Red Hat's product roadmaps but is important to > > lots of folks in the wider Fedora community, maybe that's another > > indicator we can useor indeed, if it turns out not many folks > > actually care. > > As an on the record btrfs skeptic, I think it would help to have an > end-to-end picture of what's missing or needs to be added across all > packages. If someone comes to me today and says "I want to help btrfs be > successful in Fedora", I'd like to know where to point them besides just > suggesting they get added on the btrfs kernel alias. Forgive me, but I had missed this in the sea of other messages (this thread is long and crazy now!). The Fedora Btrfs folks over the past few years have implemented the following: * Developed a DNF plugin to invoke snapper correctly for full system snapshots * Ported over the change set from SUSE to the Fedora grub2 package to support btrfs boot to snapshot * Fixed grubby to handle snapshot menu items for grub (this is kind of deprecated with BLS, though) The things that I'm looking to do: * Develop a libdnf plugin to replace the existing DNF snapper plugin (so that PackageKit is also covered) * Adapt the boot to snapshot stuff to use BLS (with boom, presumably?) The things I've been pushing off (for obvious reasons): * Enable Anaconda to allow /boot to be a btrfs subvolume (PR is in limbo...) * Ensure Anaconda configures snapper automatically when btrfs rootfs is selected * Design a sensible default btrfs layout (perhaps based on the current openSUSE one?) The mid-/long-term things I'm thinking of: * Consideration of moving the rpmdb to /usr/lib/sysimage/rpm (unifying standard Fedora and OSTree Fedora...) * A libdnf plugin to implement atomic package management using btrfs snapshots (like transactional-update, except done properly...) That's basically the full scope of the Btrfs enablement work I've planned right now. -- 真実はいつも一つ！/ Always, there's only one truth! ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote: > On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik wrote: > > > > On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote: > > > On 8/28/19 1:58 PM, Josef Bacik wrote: > > > > On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote: > > > > > On 8/26/19 11:39 PM, Neal Gompa wrote: > > > > > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott > > > > > > wrote: > > > > > > > > > > > > > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > > > > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > > > > > > > wrote: > > > > > > > > > > > > > > > > > So, there was recently a Thing where btrfs installs were > > > > > > > > > broken, and > > > > > > > > > this got accepted as a release blocker: > > > > > > > > > > > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > > > > > > > > > > > > > Summary: This bug was introduced and discovered in linux-next, > > > > > > > > it > > > > > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, > > > > > > > > patch > > > > > > > > appeared during rc1, and the patch was merged into 5.3.0-rc2. > > > > > > > > The bug > > > > > > > > resulted in a somewhat transient deadlock which caused installs > > > > > > > > to > > > > > > > > hang, but no corruption. The fix, 2 files changed, 12 > > > > > > > > insertions, 8 > > > > > > > > deletions (1/2 the insertions are comments). > > > > > > > > > > > > > > > > How remarkable or interesting is this bug? And in particular, > > > > > > > > exactly > > > > > > > > how much faster should it have been fixed in order to avoid > > > > > > > > worrying > > > > > > > > about it being a blocker bug? > > > > > > > > > > > > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > > > > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > > > > > > > 7/26 19:20 utc I confirmed upstream's patch related to this bug > > > > > > > > with > > > > > > > > upstream and updated the Fedora bug > > > > > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated > > > > > > > > the Fedora bug > > > > > > > > > > > > > > > > So in the context of status quo, where Btrfs is presented as an > > > > > > > > option > > > > > > > > in the installer and if there are bugs they Beta blocking, how > > > > > > > > could > > > > > > > > or should this have been fixed sooner? What about the handling > > > > > > > > should > > > > > > > > have been different? > > > > > > > > > > > > > > > > > > > > > > That's a fair question. This bug actually represents how this > > > > > > > _should_ > > > > > > > work. The concern is that in the past we haven't seen a lot > > > > > > > engagement > > > > > > > in the past. Maybe today that has changed as demonstrated by this > > > > > > > thread. > > > > > > > I'm still concerned about having this be a blocker vs. just > > > > > > > keeping it > > > > > > > as an option, simply because a blocker stops the entire release > > > > > > > and it > > > > > > > can be a last minute scramble to get things fixed. This was the > > > > > > > ideal > > > > > > > case for a blocker bugs and I'm skeptical about all bugs going > > > > > > > this well. > > > > > > > If we had a few more people who were willing to be on the btrfs > > > > > > > alias and > > > > > > > do the work for blocker bugs it would be a much stronger case. > > > > > > > > > > > > > > > > > > > Out of curiosity, how many such issues have we had in the past 2 > > > > > > years? I personally can't recall any monumental occasions where > > > > > > people > > > > > > were scrambling over *Btrfs* in Fedora. If anything, we continue to > > > > > > inherit the work that SUSE and Facebook are doing upstream as part > > > > > > of > > > > > > us continually updating our kernels, which I'm grateful for. > > > > > > > > > > > > And in the instances where we've had such issues, has anyone reached > > > > > > out to btrfs folks in Fedora? Chris and myself are the current ones, > > > > > > but there have been others in the past. Both of us are subscribed to > > > > > > the linux-btrfs mailing list, and Chris has a decent rapport with > > > > > > most > > > > > > of the btrfs developers. > > > > > > > > > > > > What more do you want? Actual btrfs developers in Fedora? We don't > > > > > > have any for the majority of filesystems Fedora supports, only XFS. > > > > > > Is > > > > > > there some kind of problem with communicating with the upstream > > > > > > kernel > > > > > > developers about Fedora bugs that I'm not aware of? > > > > > > > > > > > > > > > > Again, it's about length of overall development. ext and XFS have > > > > > a much longer history in general which is something that's important > > > > > for file system stability in general. It's also a bit of a catch-22 > > > > > where the rate of btrfs use in Fedora is so low we don't actually > > > > > see issues. > > > > > > > > > > > > > I note here that ext2 and ext3 are offe
Re: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik wrote: > > On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote: > > On 8/28/19 1:58 PM, Josef Bacik wrote: > > > On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote: > > > > On 8/26/19 11:39 PM, Neal Gompa wrote: > > > > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott > > > > > wrote: > > > > > > > > > > > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > > > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > > > > > > wrote: > > > > > > > > > > > > > > > So, there was recently a Thing where btrfs installs were > > > > > > > > broken, and > > > > > > > > this got accepted as a release blocker: > > > > > > > > > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > > > > > > > > > > > Summary: This bug was introduced and discovered in linux-next, it > > > > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > > > > > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The > > > > > > > bug > > > > > > > resulted in a somewhat transient deadlock which caused installs to > > > > > > > hang, but no corruption. The fix, 2 files changed, 12 insertions, > > > > > > > 8 > > > > > > > deletions (1/2 the insertions are comments). > > > > > > > > > > > > > > How remarkable or interesting is this bug? And in particular, > > > > > > > exactly > > > > > > > how much faster should it have been fixed in order to avoid > > > > > > > worrying > > > > > > > about it being a blocker bug? > > > > > > > > > > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > > > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > > > > > > 7/26 19:20 utc I confirmed upstream's patch related to this bug > > > > > > > with > > > > > > > upstream and updated the Fedora bug > > > > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated > > > > > > > the Fedora bug > > > > > > > > > > > > > > So in the context of status quo, where Btrfs is presented as an > > > > > > > option > > > > > > > in the installer and if there are bugs they Beta blocking, how > > > > > > > could > > > > > > > or should this have been fixed sooner? What about the handling > > > > > > > should > > > > > > > have been different? > > > > > > > > > > > > > > > > > > > That's a fair question. This bug actually represents how this > > > > > > _should_ > > > > > > work. The concern is that in the past we haven't seen a lot > > > > > > engagement > > > > > > in the past. Maybe today that has changed as demonstrated by this > > > > > > thread. > > > > > > I'm still concerned about having this be a blocker vs. just keeping > > > > > > it > > > > > > as an option, simply because a blocker stops the entire release and > > > > > > it > > > > > > can be a last minute scramble to get things fixed. This was the > > > > > > ideal > > > > > > case for a blocker bugs and I'm skeptical about all bugs going this > > > > > > well. > > > > > > If we had a few more people who were willing to be on the btrfs > > > > > > alias and > > > > > > do the work for blocker bugs it would be a much stronger case. > > > > > > > > > > > > > > > > Out of curiosity, how many such issues have we had in the past 2 > > > > > years? I personally can't recall any monumental occasions where people > > > > > were scrambling over *Btrfs* in Fedora. If anything, we continue to > > > > > inherit the work that SUSE and Facebook are doing upstream as part of > > > > > us continually updating our kernels, which I'm grateful for. > > > > > > > > > > And in the instances where we've had such issues, has anyone reached > > > > > out to btrfs folks in Fedora? Chris and myself are the current ones, > > > > > but there have been others in the past. Both of us are subscribed to > > > > > the linux-btrfs mailing list, and Chris has a decent rapport with most > > > > > of the btrfs developers. > > > > > > > > > > What more do you want? Actual btrfs developers in Fedora? We don't > > > > > have any for the majority of filesystems Fedora supports, only XFS. Is > > > > > there some kind of problem with communicating with the upstream kernel > > > > > developers about Fedora bugs that I'm not aware of? > > > > > > > > > > > > > Again, it's about length of overall development. ext and XFS have > > > > a much longer history in general which is something that's important > > > > for file system stability in general. It's also a bit of a catch-22 > > > > where the rate of btrfs use in Fedora is so low we don't actually > > > > see issues. > > > > > > > > > > > I note here that ext2 and ext3 are offered as file systems in > > > > > > > Custom/Advanced partitioning and in this sense have parity with > > > > > > > Btrfs. > > > > > > > If this same bug occurred in ext2 or ext3 would or should that > > > > > > > cause > > > > > > > discussion to drop them from the installer, even if the bug were > > > > > > > fixed > > > > > > > within 24 hours of discovery an
Re: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote: > On 8/28/19 1:58 PM, Josef Bacik wrote: > > On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote: > > > On 8/26/19 11:39 PM, Neal Gompa wrote: > > > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott > > > > wrote: > > > > > > > > > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > > > > > wrote: > > > > > > > > > > > > > So, there was recently a Thing where btrfs installs were broken, > > > > > > > and > > > > > > > this got accepted as a release blocker: > > > > > > > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > > > > > > > > > Summary: This bug was introduced and discovered in linux-next, it > > > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > > > > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The > > > > > > bug > > > > > > resulted in a somewhat transient deadlock which caused installs to > > > > > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > > > > > > deletions (1/2 the insertions are comments). > > > > > > > > > > > > How remarkable or interesting is this bug? And in particular, > > > > > > exactly > > > > > > how much faster should it have been fixed in order to avoid worrying > > > > > > about it being a blocker bug? > > > > > > > > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > > > > > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > > > > > > upstream and updated the Fedora bug > > > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the > > > > > > Fedora bug > > > > > > > > > > > > So in the context of status quo, where Btrfs is presented as an > > > > > > option > > > > > > in the installer and if there are bugs they Beta blocking, how could > > > > > > or should this have been fixed sooner? What about the handling > > > > > > should > > > > > > have been different? > > > > > > > > > > > > > > > > That's a fair question. This bug actually represents how this _should_ > > > > > work. The concern is that in the past we haven't seen a lot engagement > > > > > in the past. Maybe today that has changed as demonstrated by this > > > > > thread. > > > > > I'm still concerned about having this be a blocker vs. just keeping it > > > > > as an option, simply because a blocker stops the entire release and it > > > > > can be a last minute scramble to get things fixed. This was the ideal > > > > > case for a blocker bugs and I'm skeptical about all bugs going this > > > > > well. > > > > > If we had a few more people who were willing to be on the btrfs alias > > > > > and > > > > > do the work for blocker bugs it would be a much stronger case. > > > > > > > > > > > > > Out of curiosity, how many such issues have we had in the past 2 > > > > years? I personally can't recall any monumental occasions where people > > > > were scrambling over *Btrfs* in Fedora. If anything, we continue to > > > > inherit the work that SUSE and Facebook are doing upstream as part of > > > > us continually updating our kernels, which I'm grateful for. > > > > > > > > And in the instances where we've had such issues, has anyone reached > > > > out to btrfs folks in Fedora? Chris and myself are the current ones, > > > > but there have been others in the past. Both of us are subscribed to > > > > the linux-btrfs mailing list, and Chris has a decent rapport with most > > > > of the btrfs developers. > > > > > > > > What more do you want? Actual btrfs developers in Fedora? We don't > > > > have any for the majority of filesystems Fedora supports, only XFS. Is > > > > there some kind of problem with communicating with the upstream kernel > > > > developers about Fedora bugs that I'm not aware of? > > > > > > > > > > Again, it's about length of overall development. ext and XFS have > > > a much longer history in general which is something that's important > > > for file system stability in general. It's also a bit of a catch-22 > > > where the rate of btrfs use in Fedora is so low we don't actually > > > see issues. > > > > > > > > > I note here that ext2 and ext3 are offered as file systems in > > > > > > Custom/Advanced partitioning and in this sense have parity with > > > > > > Btrfs. > > > > > > If this same bug occurred in ext2 or ext3 would or should that cause > > > > > > discussion to drop them from the installer, even if the bug were > > > > > > fixed > > > > > > within 24 hours of discovery and patch? What about vfat? That's > > > > > > literally the only truly required filesystem that must work, for the > > > > > > most commonly supported hardware so it can't be dropped, we'd just > > > > > > be > > > > > > stuck until it got fixed. That work would have to be done upstream, > > > > > > yes? > > > > > > > > > > > > > > > > I don't think that
Re: Discussion: what would not blocking on btrfs look like?
On 8/28/19 1:58 PM, Josef Bacik wrote: On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote: On 8/26/19 11:39 PM, Neal Gompa wrote: On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott wrote: On 8/23/19 9:00 PM, Chris Murphy wrote: On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson wrote: So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker: https://bugzilla.redhat.com/show_bug.cgi?id=1733388 Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments). How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug? 7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different? That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case. Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for. And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers. What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of? Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues. I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes? I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases. Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat. There
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote: > On 8/26/19 11:39 PM, Neal Gompa wrote: > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott wrote: > > > > > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > > > wrote: > > > > > > > > > So, there was recently a Thing where btrfs installs were broken, and > > > > > this got accepted as a release blocker: > > > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > > > > > Summary: This bug was introduced and discovered in linux-next, it > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug > > > > resulted in a somewhat transient deadlock which caused installs to > > > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > > > > deletions (1/2 the insertions are comments). > > > > > > > > How remarkable or interesting is this bug? And in particular, exactly > > > > how much faster should it have been fixed in order to avoid worrying > > > > about it being a blocker bug? > > > > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > > > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > > > > upstream and updated the Fedora bug > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the > > > > Fedora bug > > > > > > > > So in the context of status quo, where Btrfs is presented as an option > > > > in the installer and if there are bugs they Beta blocking, how could > > > > or should this have been fixed sooner? What about the handling should > > > > have been different? > > > > > > > > > > That's a fair question. This bug actually represents how this _should_ > > > work. The concern is that in the past we haven't seen a lot engagement > > > in the past. Maybe today that has changed as demonstrated by this thread. > > > I'm still concerned about having this be a blocker vs. just keeping it > > > as an option, simply because a blocker stops the entire release and it > > > can be a last minute scramble to get things fixed. This was the ideal > > > case for a blocker bugs and I'm skeptical about all bugs going this well. > > > If we had a few more people who were willing to be on the btrfs alias and > > > do the work for blocker bugs it would be a much stronger case. > > > > > > > Out of curiosity, how many such issues have we had in the past 2 > > years? I personally can't recall any monumental occasions where people > > were scrambling over *Btrfs* in Fedora. If anything, we continue to > > inherit the work that SUSE and Facebook are doing upstream as part of > > us continually updating our kernels, which I'm grateful for. > > > > And in the instances where we've had such issues, has anyone reached > > out to btrfs folks in Fedora? Chris and myself are the current ones, > > but there have been others in the past. Both of us are subscribed to > > the linux-btrfs mailing list, and Chris has a decent rapport with most > > of the btrfs developers. > > > > What more do you want? Actual btrfs developers in Fedora? We don't > > have any for the majority of filesystems Fedora supports, only XFS. Is > > there some kind of problem with communicating with the upstream kernel > > developers about Fedora bugs that I'm not aware of? > > > > Again, it's about length of overall development. ext and XFS have > a much longer history in general which is something that's important > for file system stability in general. It's also a bit of a catch-22 > where the rate of btrfs use in Fedora is so low we don't actually > see issues. > > > > > I note here that ext2 and ext3 are offered as file systems in > > > > Custom/Advanced partitioning and in this sense have parity with Btrfs. > > > > If this same bug occurred in ext2 or ext3 would or should that cause > > > > discussion to drop them from the installer, even if the bug were fixed > > > > within 24 hours of discovery and patch? What about vfat? That's > > > > literally the only truly required filesystem that must work, for the > > > > most commonly supported hardware so it can't be dropped, we'd just be > > > > stuck until it got fixed. That work would have to be done upstream, > > > > yes? > > > > > > > > > > I don't think that's really a fair comparison. Just because options > > > are presented doesn't mean all of them are equal. ext2/ext3 and vfat > > > have been in development for much longer than btrfs and length of > > > development > > > is something that's particularly important for file system stability > > > from talking with file system developers. It's not impossible for there > > > to be bugs in ext4 for example (we've certainly seen them before) but > > > btrfs is only now gaining overall stability and we're still more likely > > > to see > > > bugs, especially with custom setups where pe
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 1:02 PM David Cantrell wrote: > > On 8/27/19 2:00 PM, Chris Murphy wrote: > > The Fedora working group's technical specification states Btrfs is to > > be the default. Yet the working group has said it's uncomfortable > > taking action on this decision expressly because the Federal kernel > > team's official recommendation is to not recommend Btrfs. And I agree. > > I trust the Fedora kernel team as they've clearly stated limited > > resources and interest in Btrfs, the expectations and parameters for > > properly supporting Btrfs either as bug blocker worthy, and as a > > default file system from a user advocacy point of view. > > OK, so 8 years has gone by and the landscape around btrfs looks > different in Fedora. Given the kernel team's position, is it worth > still having the FESCo decision and kernel team's recommendation at odds? They aren't at odds. 8 years ago FESCo decision on Btrfs 5 years ago Workstation working group decision on Btrfs (their requirements and specs docs were signed off by FESCo) 3 years ago kernel team said while they don't recommend it, they acknowledged it was ultimately up to the working group to decide, and noted they were sick of having this conversation every release The Workstation working group has, correctly in my view, put their own decision in abeyance, as a result of the kernel team's official recommendation. How does the Btrfs landscape in Fedora look different to you in three years? The way the landscape looks different to me: One, the possibility of getting a kernel engineer who works on Btrfs to join the Fedora kernel team, so that the team has the capacity to support and recommend (at least as a team, not that any individual needs to give up one tiny bit of their Btrfs scepticism) is an important development. Two, that in the interim, Red Hat is where the landscape has really changed has occurred with respect to Btrfs, and I see indications of this bias being injected back into Fedora: suggesting that a Red Hat shift away from Btrfs somehow is actually a Fedora shift away from Btrfs, suggesting a do over vote in the Workstation working group, or even an undo vote by FESCo to set aside the Workstation working group vote. And to what end? All that's needed is for the Anaconda team to provided the same courtesy of clear expectations and parameters, as the kernel team has done. > > >> If it's a best effort thing, then that makes it easier for > >> projects and contributors. Going back to Adam's original list, I would > >> suggest a FESCo decision like this should require explicit opt-in by the > >> user to enable btrfs functionality in the application in question. For > >> example, in the installer that could be enabled via a boot parameter (we > >> did this initially when btrfs functionality was first enabled in anaconda). > > > > That can only be considered to be a remarkable regression, not just in > > the context of Fedora, but in the context of the top 10 linux > > distributions all of which have visible Btrfs support in their GUI > > installers. Fedora's installer being the first to make Btrfs invisible > > by default would be a remarkable first indeed. > > I'm merely offering an example scenario. > > This does illustrate a problem with expectations among users. It's > visible now, but not a priority, which leads to frustration. Just like today's kernel team, Anaconda today are not obligated to make it a priority. But qualifying what "not a priority" actually means is necessary. The question is what are the preconditions for others who will make it a priority? And whether Anaconda is going to slow walk every PR that tries to improve Btrfs support? What about simplifying the Btrfs support in Anaconda to make it easier to maintain with priority parity with other file systems? Gut all the multiple device stuff, perhaps? Btrfs can easily do quite a lot of adjustments post-install, it's one of it's most central features. Few options are mkfs only. > >> I'm not advocating one way or another for btrfs. But it seems we as a > >> project need a larger decision and policy around btrfs in general so we > >> can set expectations for users and developers. > > > > That decision and policy has already been made. Do you want it reverted? > > I guess I meant to say "FESCo needs to revisit this decision for a > potential change". I can't parse this. Which decision, and what potential change? -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
References: FESCo, make Btrfs the default https://meetbot.fedoraproject.org/fedora-meeting/2011-06-08/fesco.2011-06-08-17.30.log.html Workstation working group's technical specification, make btrfs default https://fedoraproject.org/wiki/Workstation/Technical_Specification A point of reference that when LVM thinp was made release blocking, it was stated that usability issues are to be treated as bugs to be worked out https://meetbot.fedoraproject.org/fedora-meeting/2013-07-24/fesco.2013-07-24-18.00.log.html File system choice is up to the working group https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/message/CXW6IYIHETPS5U7IB2P6373FJDU2UAMB/ --- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 12:00 PM Chris Murphy wrote: > The Fedora working group's technical specification states Btrfs is to > be the default. Yet the working group has said it's uncomfortable > taking action on this decision expressly because the Federal kernel > team's official recommendation is to not recommend Btrfs. And I agree. Points of clarity: - it is the Workstation working group's technical spec that says this (the Server working group decided on XFS) - I agree with Workstation working group's reticence to actually deploy Btrfs as the default file system, while the Fedora kernel team recommends against Btrfs as the default. And I would like someone on the Fedora kernel team who will recommend and support it. -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 11:25 AM David Cantrell wrote: > > The installer team rejecting btrfs patches is going to be based on their > resources to support the functionality. I would say "btrfs in Fedora" > needs a FESCo decision to set expectations and policy for the project. > Is it something that Fedora wants to offer and if so, what does that > look like? FESCo already voted 8 years ago to make Btrfs the default file system, and then allowed that to wither and become moot rather than revert the decision. Then later when the editions were created, part of Fedora.next, the decision of default file systems was handed to the working groups to decide. And the Fedora kernel team has also said this is a working group decision. The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree. I trust the Fedora kernel team as they've clearly stated limited resources and interest in Btrfs, the expectations and parameters for properly supporting Btrfs either as bug blocker worthy, and as a default file system from a user advocacy point of view. > If it's a best effort thing, then that makes it easier for > projects and contributors. Going back to Adam's original list, I would > suggest a FESCo decision like this should require explicit opt-in by the > user to enable btrfs functionality in the application in question. For > example, in the installer that could be enabled via a boot parameter (we > did this initially when btrfs functionality was first enabled in anaconda). That can only be considered to be a remarkable regression, not just in the context of Fedora, but in the context of the top 10 linux distributions all of which have visible Btrfs support in their GUI installers. Fedora's installer being the first to make Btrfs invisible by default would be a remarkable first indeed. > I'm not advocating one way or another for btrfs. But it seems we as a > project need a larger decision and policy around btrfs in general so we > can set expectations for users and developers. That decision and policy has already been made. Do you want it reverted? -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 7:49 AM Samantha Bueno wrote: > > On Fri, Aug 23, 2019 at 9:17 PM Adam Williamson > wrote: > > So...what should we do? Here are the options as I see 'em: > > > > 1. Keep supporting btrfs > > 2. Just modify the criterion with a btrfs exception, even if it's weird > > 3. Rewrite the criterion entirely > > 4. Keep btrfs support in the installer (and blivet-gui) but hide it as > > we used to - require a special boot argument for it to be visible > > 5. Drop btrfs support from the installer > This thread has diverged wildly into a lot of delightful > invective-slinging at my team. In the interest of getting back to the > original topic at hand: I would be in support of three -- five. This clearly means Btrfs related bugs in Fedora's Anaconda will not block release, and by extension Btrfs could not be the default file system either. "Btrfs is not a priority" does not at all answer the central question of what level of support and resources are expected to exist in the Fedora community to maintain Btrfs in both the bug blocking, and default file system contexts. I think Laura Abbott's emails on the kernel side are quite clear ground rules for serious Btrfs bugs to remain blocker worthy, along with a path to discuss any additional expectations for Btrfs to be a default file system in some capacity. Unqualified statements like "it is safe to say btrfs will not be the default" and "Btrfs is not a priority" and this 'zero chance Btrfs' comment [1] are completely compatible with assuming that no matter how much work is done by others, those things will not appear in Anaconda because the decision has been made, it is a fait accompli. It is very important to have clarity exactly to what degree Btrfs is not a priority. If the Anaconda team cannot state the terms and expectations for the #1 option in Adam's list, that's troubling. Again, I think it's completely fine if Red Hat Anaconda folks are disinterested in Btrfs, but it's a very different thing if there's resistance to it, and I'm getting a lot of language that is compatible with resisting Btrfs no matter what. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c10 -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 8:57 AM Josh Boyer wrote: > > On Tue, Aug 27, 2019 at 8:48 AM Neal Gompa wrote: > > > > On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer > > wrote: > > > > > > On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa wrote: > > > > > > > > On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer > > > > wrote: > > > > > > > > > > On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > > > > > > > > > > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > > > > > > > > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > > > > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > > > > > > > > > > > I understand them. The point is, for them and even us (the > > > > > > > > > installer) > > > > > > > > > is work on BTRFS not a priority. It's something we can't > > > > > > > > > benefit on > > > > > > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > > > > > > solution. > > > > > > > > > However, it still giving us bugs and making our test surface > > > > > > > > > bigger. > > > > > > > > > > > > > > > > > > > From the Anaconda team PoV it would make our lives easier > > > > > > > > > > to not > > > > > > > > > support BTRFS at all. I'm not saying that we should drop > > > > > > > > > BTRFS in > > > > > > > > > Fedora, only that it would be easier for Anaconda team to be > > > > > > > > > without > > > > > > > > > that on Fedora. > > > > > > > > > > > > > > > > This is flat-out a trap. This is what makes Anaconda such a > > > > > > > > failure > > > > > > > > as > > > > > > > > a community project. Why does the past (RHEL) affect the > > > > > > > > present and > > > > > > > > future (Fedora)? There's basically no way whatsoever to make > > > > > > > > anything > > > > > > > > better with this logic. The Anaconda releases that any > > > > > > > > improvements > > > > > > > > would be going into aren't even landing into the RHEL 8 branch > > > > > > > > that > > > > > > > > governs the latest iteration of Fedora's past. From any > > > > > > > > reasonable > > > > > > > > person's perspective, this answer makes no sense unless you're > > > > > > > > using > > > > > > > > RHEL as an excuse to not support Fedora. > > > > > > > > > > > > > > > > > > > > > > RHEL is not the past. Everything we do we have to think that it > > > > > > > will go > > > > > > > to RHEL and if it is Fedora specific we have to create a way to > > > > > > > disable > > > > > > > the functionality for another RHEL branching. And yes, we have a > > > > > > > few > > > > > > > things (not only a BTRFS) specific to Fedora the same way as a few > > > > > > > things specific to RHEL which are disabled on Fedora. > > > > > > > > > > > > > > And as I wrote before, I'm not saying that we will remove the > > > > > > > BTRFS > > > > > > > support from Fedora. The point is that making the list specific to > > > > > > > releases smaller will make our live easier. > > > > > > > > > > > > > > > > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > > > > > > from an old version of Fedora that's not supported anymore. It is > > > > > > the > > > > > > result of decisions that aren't supposed to apply to Fedora. And it > > > > > > is > > > > > > the result of a different bias that should never apply to Fedora if > > > > > > the RH ecosystem is supposed to be able to evolve. > > > > > > > > > > There is always the *next* RHEL. Or, if you want to remove a product > > > > > context, the next enterprise operating system derived from Fedora. > > > > > RHEL/enterprise is both past and future and Fedora focuses on the > > > > > future. You cannot dismiss enterprise as a target by waiving it away > > > > > as "past". > > > > > > > > > > > > > Until there's a branch in Anaconda's git for the next version of RHEL, > > > > it doesn't exist yet. I'm sure people are *thinking* about it, but > > > > it's obviously on the back burner for a little while. I would expect > > > > to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now > > > > it doesn't exist. > > > > > > In 1.5 years is entirely too late. Massively so. I know people think > > > we're paying lip-service to upstream when discussing Fedora and RHEL, > > > but even with a terrible "import once" model the code being developed > > > in Fedora right now will materially land in RHEL 9. Your presumption > > > on a branch needing to exist is simply incorrect. > > > > > > > For us non RH people, there's nothing for us to do about RHEL 9 right > > now. Your planning is done in secret, and there's little to no > > community feedback loop for RHEL.next. I agree that in ordinary > > circumstances, it's too late once the branch has happened, because > > usually that means it's the stabilizing phase. But there's nothing I > > can do before then. > > I think that's a problem and one which we're trying to address. > However, whether code lands due to internal planning or due to > community contribution, it all still impacts any futu
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 8:48 AM Neal Gompa wrote: > > On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer wrote: > > > > On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa wrote: > > > > > > On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer > > > wrote: > > > > > > > > On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > > > > > > > > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > > > > > > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > > > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > > > > > > > > > I understand them. The point is, for them and even us (the > > > > > > > > installer) > > > > > > > > is work on BTRFS not a priority. It's something we can't > > > > > > > > benefit on > > > > > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > > > > > solution. > > > > > > > > However, it still giving us bugs and making our test surface > > > > > > > > bigger. > > > > > > > > > > > > > > > > > From the Anaconda team PoV it would make our lives easier to > > > > > > > > > not > > > > > > > > support BTRFS at all. I'm not saying that we should drop BTRFS > > > > > > > > in > > > > > > > > Fedora, only that it would be easier for Anaconda team to be > > > > > > > > without > > > > > > > > that on Fedora. > > > > > > > > > > > > > > This is flat-out a trap. This is what makes Anaconda such a > > > > > > > failure > > > > > > > as > > > > > > > a community project. Why does the past (RHEL) affect the present > > > > > > > and > > > > > > > future (Fedora)? There's basically no way whatsoever to make > > > > > > > anything > > > > > > > better with this logic. The Anaconda releases that any > > > > > > > improvements > > > > > > > would be going into aren't even landing into the RHEL 8 branch > > > > > > > that > > > > > > > governs the latest iteration of Fedora's past. From any reasonable > > > > > > > person's perspective, this answer makes no sense unless you're > > > > > > > using > > > > > > > RHEL as an excuse to not support Fedora. > > > > > > > > > > > > > > > > > > > RHEL is not the past. Everything we do we have to think that it > > > > > > will go > > > > > > to RHEL and if it is Fedora specific we have to create a way to > > > > > > disable > > > > > > the functionality for another RHEL branching. And yes, we have a few > > > > > > things (not only a BTRFS) specific to Fedora the same way as a few > > > > > > things specific to RHEL which are disabled on Fedora. > > > > > > > > > > > > And as I wrote before, I'm not saying that we will remove the BTRFS > > > > > > support from Fedora. The point is that making the list specific to > > > > > > releases smaller will make our live easier. > > > > > > > > > > > > > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > > > > > from an old version of Fedora that's not supported anymore. It is the > > > > > result of decisions that aren't supposed to apply to Fedora. And it is > > > > > the result of a different bias that should never apply to Fedora if > > > > > the RH ecosystem is supposed to be able to evolve. > > > > > > > > There is always the *next* RHEL. Or, if you want to remove a product > > > > context, the next enterprise operating system derived from Fedora. > > > > RHEL/enterprise is both past and future and Fedora focuses on the > > > > future. You cannot dismiss enterprise as a target by waiving it away > > > > as "past". > > > > > > > > > > Until there's a branch in Anaconda's git for the next version of RHEL, > > > it doesn't exist yet. I'm sure people are *thinking* about it, but > > > it's obviously on the back burner for a little while. I would expect > > > to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now > > > it doesn't exist. > > > > In 1.5 years is entirely too late. Massively so. I know people think > > we're paying lip-service to upstream when discussing Fedora and RHEL, > > but even with a terrible "import once" model the code being developed > > in Fedora right now will materially land in RHEL 9. Your presumption > > on a branch needing to exist is simply incorrect. > > > > For us non RH people, there's nothing for us to do about RHEL 9 right > now. Your planning is done in secret, and there's little to no > community feedback loop for RHEL.next. I agree that in ordinary > circumstances, it's too late once the branch has happened, because > usually that means it's the stabilizing phase. But there's nothing I > can do before then. I think that's a problem and one which we're trying to address. However, whether code lands due to internal planning or due to community contribution, it all still impacts any future OS release, Fedora and enterprise alike. > > > > > From the way you describe it, Fedora is just something occasionally > > > > > give lip service to while your main focus is RHEL. That's fine, but > > > > > that is a problem for the Fedora context. > > > > > > > > Neal, I don't understand. The source code to anaconda is a
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer wrote: > > On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa wrote: > > > > On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer > > wrote: > > > > > > On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > > > > > > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > > > > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > > > > > > > I understand them. The point is, for them and even us (the > > > > > > > installer) > > > > > > > is work on BTRFS not a priority. It's something we can't benefit > > > > > > > on > > > > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > > > > solution. > > > > > > > However, it still giving us bugs and making our test surface > > > > > > > bigger. > > > > > > > > > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > > > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > > > > > Fedora, only that it would be easier for Anaconda team to be > > > > > > > without > > > > > > > that on Fedora. > > > > > > > > > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > > > > > as > > > > > > a community project. Why does the past (RHEL) affect the present and > > > > > > future (Fedora)? There's basically no way whatsoever to make > > > > > > anything > > > > > > better with this logic. The Anaconda releases that any improvements > > > > > > would be going into aren't even landing into the RHEL 8 branch that > > > > > > governs the latest iteration of Fedora's past. From any reasonable > > > > > > person's perspective, this answer makes no sense unless you're using > > > > > > RHEL as an excuse to not support Fedora. > > > > > > > > > > > > > > > > RHEL is not the past. Everything we do we have to think that it will > > > > > go > > > > > to RHEL and if it is Fedora specific we have to create a way to > > > > > disable > > > > > the functionality for another RHEL branching. And yes, we have a few > > > > > things (not only a BTRFS) specific to Fedora the same way as a few > > > > > things specific to RHEL which are disabled on Fedora. > > > > > > > > > > And as I wrote before, I'm not saying that we will remove the BTRFS > > > > > support from Fedora. The point is that making the list specific to > > > > > releases smaller will make our live easier. > > > > > > > > > > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > > > > from an old version of Fedora that's not supported anymore. It is the > > > > result of decisions that aren't supposed to apply to Fedora. And it is > > > > the result of a different bias that should never apply to Fedora if > > > > the RH ecosystem is supposed to be able to evolve. > > > > > > There is always the *next* RHEL. Or, if you want to remove a product > > > context, the next enterprise operating system derived from Fedora. > > > RHEL/enterprise is both past and future and Fedora focuses on the > > > future. You cannot dismiss enterprise as a target by waiving it away > > > as "past". > > > > > > > Until there's a branch in Anaconda's git for the next version of RHEL, > > it doesn't exist yet. I'm sure people are *thinking* about it, but > > it's obviously on the back burner for a little while. I would expect > > to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now > > it doesn't exist. > > In 1.5 years is entirely too late. Massively so. I know people think > we're paying lip-service to upstream when discussing Fedora and RHEL, > but even with a terrible "import once" model the code being developed > in Fedora right now will materially land in RHEL 9. Your presumption > on a branch needing to exist is simply incorrect. > For us non RH people, there's nothing for us to do about RHEL 9 right now. Your planning is done in secret, and there's little to no community feedback loop for RHEL.next. I agree that in ordinary circumstances, it's too late once the branch has happened, because usually that means it's the stabilizing phase. But there's nothing I can do before then. > > > > From the way you describe it, Fedora is just something occasionally > > > > give lip service to while your main focus is RHEL. That's fine, but > > > > that is a problem for the Fedora context. > > > > > > Neal, I don't understand. The source code to anaconda is available. > > > What is preventing you from taking it and making a micro-fork that > > > does better btrfs enablement, and packaging that in Fedora and using > > > it in a spin? > > > > > > > I have seriously contemplated it. It isn't the first time I've done a > > fork because I had to[1]. > > > > But the main reason I don't do it is because it will cause more damage > > in the Fedora community by doing so. Two sets of packages for Anaconda > > that all the things that depend on Anaconda could cause a huge level > > of breakage because the two versions must *alwa
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa wrote: > > On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer wrote: > > > > On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > > > > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > > > > > I understand them. The point is, for them and even us (the > > > > > > installer) > > > > > > is work on BTRFS not a priority. It's something we can't benefit on > > > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > > > solution. > > > > > > However, it still giving us bugs and making our test surface > > > > > > bigger. > > > > > > > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > > > > Fedora, only that it would be easier for Anaconda team to be > > > > > > without > > > > > > that on Fedora. > > > > > > > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > > > > as > > > > > a community project. Why does the past (RHEL) affect the present and > > > > > future (Fedora)? There's basically no way whatsoever to make anything > > > > > better with this logic. The Anaconda releases that any improvements > > > > > would be going into aren't even landing into the RHEL 8 branch that > > > > > governs the latest iteration of Fedora's past. From any reasonable > > > > > person's perspective, this answer makes no sense unless you're using > > > > > RHEL as an excuse to not support Fedora. > > > > > > > > > > > > > RHEL is not the past. Everything we do we have to think that it will go > > > > to RHEL and if it is Fedora specific we have to create a way to disable > > > > the functionality for another RHEL branching. And yes, we have a few > > > > things (not only a BTRFS) specific to Fedora the same way as a few > > > > things specific to RHEL which are disabled on Fedora. > > > > > > > > And as I wrote before, I'm not saying that we will remove the BTRFS > > > > support from Fedora. The point is that making the list specific to > > > > releases smaller will make our live easier. > > > > > > > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > > > from an old version of Fedora that's not supported anymore. It is the > > > result of decisions that aren't supposed to apply to Fedora. And it is > > > the result of a different bias that should never apply to Fedora if > > > the RH ecosystem is supposed to be able to evolve. > > > > There is always the *next* RHEL. Or, if you want to remove a product > > context, the next enterprise operating system derived from Fedora. > > RHEL/enterprise is both past and future and Fedora focuses on the > > future. You cannot dismiss enterprise as a target by waiving it away > > as "past". > > > > Until there's a branch in Anaconda's git for the next version of RHEL, > it doesn't exist yet. I'm sure people are *thinking* about it, but > it's obviously on the back burner for a little while. I would expect > to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now > it doesn't exist. In 1.5 years is entirely too late. Massively so. I know people think we're paying lip-service to upstream when discussing Fedora and RHEL, but even with a terrible "import once" model the code being developed in Fedora right now will materially land in RHEL 9. Your presumption on a branch needing to exist is simply incorrect. > > > From the way you describe it, Fedora is just something occasionally > > > give lip service to while your main focus is RHEL. That's fine, but > > > that is a problem for the Fedora context. > > > > Neal, I don't understand. The source code to anaconda is available. > > What is preventing you from taking it and making a micro-fork that > > does better btrfs enablement, and packaging that in Fedora and using > > it in a spin? > > > > I have seriously contemplated it. It isn't the first time I've done a > fork because I had to[1]. > > But the main reason I don't do it is because it will cause more damage > in the Fedora community by doing so. Two sets of packages for Anaconda > that all the things that depend on Anaconda could cause a huge level > of breakage because the two versions must *always* be drop-in > replacements for each other. If they're not, it makes it impossible to > leverage the Fedora tooling to do things like making spins and such. I > don't even *know* what kind of work it would entail if I wanted it to > be an official spin composed through pungi. I'm pretty sure the releng > folks would kill me for doing that, as now pungi would have be aware > and switch anaconda packages for lorax... So... more time investment. Yep. > Additionally, it would require a fork of pykickstart so that further > enhancements to the Btrfs partitioning can be defined. However, *that* > causes bigger problems because now t
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer wrote: > > On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > > > I understand them. The point is, for them and even us (the > > > > > installer) > > > > > is work on BTRFS not a priority. It's something we can't benefit on > > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > > solution. > > > > > However, it still giving us bugs and making our test surface > > > > > bigger. > > > > > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > > > Fedora, only that it would be easier for Anaconda team to be > > > > > without > > > > > that on Fedora. > > > > > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > > > as > > > > a community project. Why does the past (RHEL) affect the present and > > > > future (Fedora)? There's basically no way whatsoever to make anything > > > > better with this logic. The Anaconda releases that any improvements > > > > would be going into aren't even landing into the RHEL 8 branch that > > > > governs the latest iteration of Fedora's past. From any reasonable > > > > person's perspective, this answer makes no sense unless you're using > > > > RHEL as an excuse to not support Fedora. > > > > > > > > > > RHEL is not the past. Everything we do we have to think that it will go > > > to RHEL and if it is Fedora specific we have to create a way to disable > > > the functionality for another RHEL branching. And yes, we have a few > > > things (not only a BTRFS) specific to Fedora the same way as a few > > > things specific to RHEL which are disabled on Fedora. > > > > > > And as I wrote before, I'm not saying that we will remove the BTRFS > > > support from Fedora. The point is that making the list specific to > > > releases smaller will make our live easier. > > > > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > > from an old version of Fedora that's not supported anymore. It is the > > result of decisions that aren't supposed to apply to Fedora. And it is > > the result of a different bias that should never apply to Fedora if > > the RH ecosystem is supposed to be able to evolve. > > There is always the *next* RHEL. Or, if you want to remove a product > context, the next enterprise operating system derived from Fedora. > RHEL/enterprise is both past and future and Fedora focuses on the > future. You cannot dismiss enterprise as a target by waiving it away > as "past". > Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist. > > From the way you describe it, Fedora is just something occasionally > > give lip service to while your main focus is RHEL. That's fine, but > > that is a problem for the Fedora context. > > Neal, I don't understand. The source code to anaconda is available. > What is preventing you from taking it and making a micro-fork that > does better btrfs enablement, and packaging that in Fedora and using > it in a spin? > I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1]. But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax... Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files). I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so. [1]: https://lists.fedoraproject.org/archives/list/liv...@lists.fedoraproject.org/thread/VL666ET5FR6MTSGGTHDULDSQN5DEWUUM/ > My guess would be that you perha
Re: Discussion: what would not blocking on btrfs look like?
On 8/26/19 11:39 PM, Neal Gompa wrote: On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott wrote: On 8/23/19 9:00 PM, Chris Murphy wrote: On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson wrote: So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker: https://bugzilla.redhat.com/show_bug.cgi?id=1733388 Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments). How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug? 7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different? That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case. Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for. And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers. What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of? Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues. I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes? I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases. Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat. > There are plenty of examples of Red Hat doing crazy/experimental things... I'd like to think Red Hat isn't
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa wrote: > > On Tue, Aug 27, 2019 at 5:55 AM wrote: > > > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > > > I understand them. The point is, for them and even us (the > > > > installer) > > > > is work on BTRFS not a priority. It's something we can't benefit on > > > > RHEL and it could be almost completely replaced by LVM + xfs > > > > solution. > > > > However, it still giving us bugs and making our test surface > > > > bigger. > > > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > > Fedora, only that it would be easier for Anaconda team to be > > > > without > > > > that on Fedora. > > > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > > as > > > a community project. Why does the past (RHEL) affect the present and > > > future (Fedora)? There's basically no way whatsoever to make anything > > > better with this logic. The Anaconda releases that any improvements > > > would be going into aren't even landing into the RHEL 8 branch that > > > governs the latest iteration of Fedora's past. From any reasonable > > > person's perspective, this answer makes no sense unless you're using > > > RHEL as an excuse to not support Fedora. > > > > > > > RHEL is not the past. Everything we do we have to think that it will go > > to RHEL and if it is Fedora specific we have to create a way to disable > > the functionality for another RHEL branching. And yes, we have a few > > things (not only a BTRFS) specific to Fedora the same way as a few > > things specific to RHEL which are disabled on Fedora. > > > > And as I wrote before, I'm not saying that we will remove the BTRFS > > support from Fedora. The point is that making the list specific to > > releases smaller will make our live easier. > > > > By definition, RHEL *is* the past from a Fedora context. It's forked > from an old version of Fedora that's not supported anymore. It is the > result of decisions that aren't supposed to apply to Fedora. And it is > the result of a different bias that should never apply to Fedora if > the RH ecosystem is supposed to be able to evolve. There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past". > From the way you describe it, Fedora is just something occasionally > give lip service to while your main focus is RHEL. That's fine, but > that is a problem for the Fedora context. Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin? My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done. Before you think I'm being naive or disingenuous, I'm truly not. I understand the frustration of wanting to get code into a project or product that isn't willing to take that code. However, it's not only about the code. The code review and acceptance isn't the end of the time investment. It is the start. There's test cases, test infrastructure, debugging support issues, on-going code changes and refactoring, etc etc. One of the ways to limit these impacts is to be selective in what enablement you choose to bring into a project. That is what the anaconda team is doing here. The beauty (and ugly!) of open source is that you can still take the code and do what you want if you are willing to make that investment. josh ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 5:55 AM wrote: > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > On Mon, Aug 26, 2019 at 7:16 AM wrote: > > > > > > I understand them. The point is, for them and even us (the > > > installer) > > > is work on BTRFS not a priority. It's something we can't benefit on > > > RHEL and it could be almost completely replaced by LVM + xfs > > > solution. > > > However, it still giving us bugs and making our test surface > > > bigger. > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > Fedora, only that it would be easier for Anaconda team to be > > > without > > > that on Fedora. > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > as > > a community project. Why does the past (RHEL) affect the present and > > future (Fedora)? There's basically no way whatsoever to make anything > > better with this logic. The Anaconda releases that any improvements > > would be going into aren't even landing into the RHEL 8 branch that > > governs the latest iteration of Fedora's past. From any reasonable > > person's perspective, this answer makes no sense unless you're using > > RHEL as an excuse to not support Fedora. > > > > RHEL is not the past. Everything we do we have to think that it will go > to RHEL and if it is Fedora specific we have to create a way to disable > the functionality for another RHEL branching. And yes, we have a few > things (not only a BTRFS) specific to Fedora the same way as a few > things specific to RHEL which are disabled on Fedora. > > And as I wrote before, I'm not saying that we will remove the BTRFS > support from Fedora. The point is that making the list specific to > releases smaller will make our live easier. > By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve. From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context. -- 真実はいつも一つ！/ Always, there's only one truth! ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Mon, Aug 26, 2019 at 5:16 AM wrote: > I understand them. The point is, for them and even us (the installer) > is work on BTRFS not a priority. It's something we can't benefit on > RHEL and it could be almost completely replaced by LVM + xfs solution. > However, it still giving us bugs and making our test surface bigger. > > >From the Anaconda team PoV it would make our lives easier to not > support BTRFS at all. I'm not saying that we should drop BTRFS in > Fedora, only that it would be easier for Anaconda team to be without > that on Fedora. It would also be easier if Custom and Advanced partitioning UIs were dropped entirely. Most Linux distros now support Btrfs. All the top 10 do. One, currently ranked #7 Solus, supports it via a point and shoot installer, deferring to Gparted to actually set it up. All the others have a custom interface that supports Btrfs directly. Meanwhile, #23 Parrot uses Btrfs by default for home and root. And so does openSUSE for a while now. And the idea being floated, is that Fedora shouldn't have a sense of adventure, but to maybe drop Btrfs from the installer. Fedora would be the first, if it did. It is completely reasonable for Red Hat to have maintainability concerns about Btrfs for RHEL, and it's entirely fair for Red Hat to have a bias against it. If it were true that Red Hat is, however unintentionally, injecting its Btrfs bias into Fedora, that would be troubling. -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 2:26 PM Laura Abbott wrote: > > I don't think we need someone to join the team per se. All we need is > someone who we can assign bugs to and have them work through the issues, > whether that's development or working with upstream to test. We have > a fedora-btrfs bug alias and we can add whoever we want on here. > > I'm okay with keeping btrfs alive if there's enough of a community who > is willing to actually fix bugs and work through the issues. We > do this with other parts of the kernel too. Past Fedora kernel team statements officially recommended against Btrfs. I think it would be weird to make Btrfs the default file system, were that still the case. And one way to alleviate that, it sounds like, is if there were a Btrfs developer on the Fedora kernel team in some capacity, even if it is strictly Btrfs bugs. But I'm open to other ideas. And if it's just a case of release criteria violating Btrfs bugs remaining blocker worthy, then I think it can go either way. > I think 3-5 are the best options right now with a focus on having btrfs > be available but not "supported". If we had a group of people who were > willing to actively debug issues like the one Adam reported, I'd be okay > with #1. I'm on the fedora-kernel-btrfs@ since February, and also supposedly get btrfs-progs bug notifications. And I've been on linux-btrfs@ since early days, they know who I am, even though I can't code my way out of a hat. They've always been responsive when I show them bugs I can reproduce. -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Mon, Aug 26, 2019 at 7:16 AM wrote: > > On Sat, 2019-08-24 at 07:31 -0700, Adam Williamson wrote: > > On Fri, 2019-08-23 at 19:00 -0600, Chris Murphy wrote: > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > > wrote: > > > > > > > So, there was recently a Thing where btrfs installs were broken, > > > > and > > > > this got accepted as a release blocker: > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > > > Summary: This bug was introduced and discovered in linux-next, it > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The > > > bug > > > resulted in a somewhat transient deadlock which caused installs to > > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > > > deletions (1/2 the insertions are comments). > > > > > > How remarkable or interesting is this bug? And in particular, > > > exactly > > > how much faster should it have been fixed in order to avoid > > > worrying > > > about it being a blocker bug? > > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > > 7/26 19:20 utc I confirmed upstream's patch related to this bug > > > with > > > upstream and updated the Fedora bug > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the > > > Fedora bug > > > > > > So in the context of status quo, where Btrfs is presented as an > > > option > > > in the installer and if there are bugs they Beta blocking, how > > > could > > > or should this have been fixed sooner? What about the handling > > > should > > > have been different? > > > > Nothing. The kernel team's concern is not related to this bug or the > > handling of this bug in any way. The only relevance of the bug is > > that > > it alerted the kernel team to the fact that we currently block on > > btrfs, which they think we should not. > > I understand them. The point is, for them and even us (the installer) > is work on BTRFS not a priority. It's something we can't benefit on > RHEL and it could be almost completely replaced by LVM + xfs solution. > However, it still giving us bugs and making our test surface bigger. > > >From the Anaconda team PoV it would make our lives easier to not > support BTRFS at all. I'm not saying that we should drop BTRFS in > Fedora, only that it would be easier for Anaconda team to be without > that on Fedora. This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora. -- 真実はいつも一つ！/ Always, there's only one truth! ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott wrote: > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > wrote: > > > >> So, there was recently a Thing where btrfs installs were broken, and > >> this got accepted as a release blocker: > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > Summary: This bug was introduced and discovered in linux-next, it > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug > > resulted in a somewhat transient deadlock which caused installs to > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > > deletions (1/2 the insertions are comments). > > > > How remarkable or interesting is this bug? And in particular, exactly > > how much faster should it have been fixed in order to avoid worrying > > about it being a blocker bug? > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > > upstream and updated the Fedora bug > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora > > bug > > > > So in the context of status quo, where Btrfs is presented as an option > > in the installer and if there are bugs they Beta blocking, how could > > or should this have been fixed sooner? What about the handling should > > have been different? > > > > That's a fair question. This bug actually represents how this _should_ > work. The concern is that in the past we haven't seen a lot engagement > in the past. Maybe today that has changed as demonstrated by this thread. > I'm still concerned about having this be a blocker vs. just keeping it > as an option, simply because a blocker stops the entire release and it > can be a last minute scramble to get things fixed. This was the ideal > case for a blocker bugs and I'm skeptical about all bugs going this well. > If we had a few more people who were willing to be on the btrfs alias and > do the work for blocker bugs it would be a much stronger case. > Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for. And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers. What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of? > > I note here that ext2 and ext3 are offered as file systems in > > Custom/Advanced partitioning and in this sense have parity with Btrfs. > > If this same bug occurred in ext2 or ext3 would or should that cause > > discussion to drop them from the installer, even if the bug were fixed > > within 24 hours of discovery and patch? What about vfat? That's > > literally the only truly required filesystem that must work, for the > > most commonly supported hardware so it can't be dropped, we'd just be > > stuck until it got fixed. That work would have to be done upstream, > > yes? > > > > I don't think that's really a fair comparison. Just because options > are presented doesn't mean all of them are equal. ext2/ext3 and vfat > have been in development for much longer than btrfs and length of development > is something that's particularly important for file system stability > from talking with file system developers. It's not impossible for there > to be bugs in ext4 for example (we've certainly seen them before) but > btrfs is only now gaining overall stability and we're still more likely to see > bugs, especially with custom setups where people are likely to find > edge cases. > Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat. There are plenty of examples of Red Hat doing crazy/experimental things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is... -- 真実はいつも一つ！/ Always, there's only one truth! ___
Re: Discussion: what would not blocking on btrfs look like?
On Mon, Aug 26, 2019 at 10:48 PM Chris Murphy wrote: > > On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes wrote: > > > > All of this, the criteria, and the UI support for btrfs are from the > > many years old proposal to make btrfs the default filesystem. In the > > beginning, it was not ready, but did show promise. This proposal came > > up for several releases in a row, and at the end of it, even the > > upstream developers recommended against it. > > Josef Bacik alone pushed for it. And it was Fedora that wasn't ready > for Btrfs, not the other way around. In Josef's last couple emails to > devel@ he stated the decision would need to be made by others, not > him. He pretty much gave up once SUSE got there first. > Indeed. In fact, I more or less took over the reigns of trying to improve Fedora's Btrfs support after Josef gave up. > I'm not aware of any upstream developers recommending Fedora not use > it. A significant chunk of upstream are at SUSE and by this time had > moved to Btrfs by default, so it'd be a little weird if they're > recommending against the thing. > If anything, the Btrfs stack developers at SUSE have been nothing but helpful whenever I've talked to them about working on bringing this to Fedora. I've literally *never* heard them say that Btrfs isn't ready. However, they have told me in the past to be cautious with some features for "enterprise supportability". But they've never said anything about Btrfs not being ready as a whole. > > > At this point, it is safe > > to say that btrfs will not be the default. Since that time, things > > have not gotten better. > > This is ambiguous. One possible way to read this is: no matter what > resources are put into supporting it in Fedora, it's safe to say it > won't be the default. Another possibility: the support resources > necessary haven't materialized, therefore it won't be the default. > > I would like to better understand why it is "safe to say" it won't be > the default. > I really didn't want this to come up, and I ignored this the first time, but I can't anymore... I'm pretty sure this is another variation of the "shutdowns" I keep getting when trying to work on Btrfs support in Fedora. It's surprisingly annoying that Fedora as a community distribution has touch points where community members have basically zero chance at being successful at anything. Honestly, I'm surprised we've kept Anaconda as the installer for Fedora, given the attitude I and other community members get from the Anaconda developers and sheer amount of pain and agony we have to deal with to even attempt simple contributions, much less complex ones. Amazingly, I do have a single commit in Anaconda[0], but the amount of effort and time it took to get it in was ridiculous. And while I like the Anaconda installer, I don't like that there can never be a community around it. It's one of those projects that has a stranglehold around it that is designed to discourage you. And that's not even with things like Btrfs. There's been a request (with code even!) from the Qubes OS folks to add GPG key support to kickstart[1] and Anaconda[2] for literally years. I've wanted to accept the changes for livecd-tools[3] for years, but I can't until pykickstart and anaconda support it. I'm not even going to get into the repo --priority option from the FedBerry guys. By any meaningful measure, Anaconda has less community than Btrfs, and it remains the "blessed" installer. Moreover, it gets to singlehandedly dictate what the distribution supports. And its developers basically get to shut down any discussion of anything related to Btrfs, in any form or fashion[4]. And you wonder why Btrfs support is so weak in Fedora? Because of all this, I was discouraged from finishing and submitting my change proposal to fully enable Btrfs with full system snapshots[5]. I'm honestly not surprised that Josef gave up. I'm surprised that I'm still stubborn enough to try to make it happen, but perhaps there's a part of me that hates giving up... [0]: https://github.com/rhinstaller/anaconda/commit/c8c5589e4a4c5451d91ae8c47bf2fe0270d2af5f [1]: https://github.com/bcl/pykickstart/pull/32 [2]: https://github.com/rhinstaller/anaconda/pull/375 [3]: https://github.com/livecd-tools/livecd-tools/pull/14 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c9 [5]: https://fedoraproject.org/wiki/Changes/BtrfsWithFullSystemSnapshots > > > > > Yes, there is active btrfs development > > upstream. It is fairly narrowly focused, and not something we can > > rely upon for a supported default among the Fedora use cases. > > The thread is ostensibly about whether it's appropriate to block > release on Btrfs related bugs, not whether it should be the default > file system. But as it's been brought up, I'd like to know if there's > any difference in the expected support resources between it remaining > "blocker worthy" versus becoming "a default file system" somewhere in > the Fedora ecosystem in a release
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes wrote: > > All of this, the criteria, and the UI support for btrfs are from the > many years old proposal to make btrfs the default filesystem. In the > beginning, it was not ready, but did show promise. This proposal came > up for several releases in a row, and at the end of it, even the > upstream developers recommended against it. Josef Bacik alone pushed for it. And it was Fedora that wasn't ready for Btrfs, not the other way around. In Josef's last couple emails to devel@ he stated the decision would need to be made by others, not him. He pretty much gave up once SUSE got there first. I'm not aware of any upstream developers recommending Fedora not use it. A significant chunk of upstream are at SUSE and by this time had moved to Btrfs by default, so it'd be a little weird if they're recommending against the thing. > At this point, it is safe > to say that btrfs will not be the default. Since that time, things > have not gotten better. This is ambiguous. One possible way to read this is: no matter what resources are put into supporting it in Fedora, it's safe to say it won't be the default. Another possibility: the support resources necessary haven't materialized, therefore it won't be the default. I would like to better understand why it is "safe to say" it won't be the default. > > Yes, there is active btrfs development > upstream. It is fairly narrowly focused, and not something we can > rely upon for a supported default among the Fedora use cases. The thread is ostensibly about whether it's appropriate to block release on Btrfs related bugs, not whether it should be the default file system. But as it's been brought up, I'd like to know if there's any difference in the expected support resources between it remaining "blocker worthy" versus becoming "a default file system" somewhere in the Fedora ecosystem in a release blocking capacity (i.e. presumably a Fedora Spin could choose to make Btrfs its default file system, but that wouldn't be release blocking). > While > Fedora does enable it in the kernel, and plans to continue doing so, > it is enabled in the "if you break it, you get to keep the pieces" > method of many other options. Sure, we will be happy to bring in a > patch that is headed upstream if it fixes a bug, and someone points us > to it. No, we aren't going to spend time debugging issues with it > ourselves. There is no shortage of issues in more "core" kernel > pieces that require attention. That's understandable and reasonable. I don't think anyone uninterested in supporting Btrfs should be made to feel like they ought to. Life is short, do what you're interested in doing, no more. -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On 8/23/19 9:00 PM, Chris Murphy wrote: On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson wrote: So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker: https://bugzilla.redhat.com/show_bug.cgi?id=1733388 Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments). How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug? 7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different? That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case. I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes? I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases. Thanks, Laura ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 8:00 PM Chris Murphy wrote: > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > wrote: > > > So, there was recently a Thing where btrfs installs were broken, and > > this got accepted as a release blocker: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > Summary: This bug was introduced and discovered in linux-next, it > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug > resulted in a somewhat transient deadlock which caused installs to > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > deletions (1/2 the insertions are comments). > > How remarkable or interesting is this bug? And in particular, exactly > how much faster should it have been fixed in order to avoid worrying > about it being a blocker bug? > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > 7/25 22:33 utc bug was first reported in Fedora bugzilla > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > upstream and updated the Fedora bug > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug > > So in the context of status quo, where Btrfs is presented as an option > in the installer and if there are bugs they Beta blocking, how could > or should this have been fixed sooner? What about the handling should > have been different? > > I note here that ext2 and ext3 are offered as file systems in > Custom/Advanced partitioning and in this sense have parity with Btrfs. > If this same bug occurred in ext2 or ext3 would or should that cause > discussion to drop them from the installer, even if the bug were fixed > within 24 hours of discovery and patch? What about vfat? That's > literally the only truly required filesystem that must work, for the > most commonly supported hardware so it can't be dropped, we'd just be > stuck until it got fixed. That work would have to be done upstream, > yes? From my standpoint, ext4 and xfs are the primary supported root filesystems. I don't think that anything else should be release blocking. As for whether the installer exposes other options, it is really more of an installer and QA question. We are certainly not even discussing turning them off in the kernel at this point, and I don't think that we should. > -- > Chris Murphy > ___ > kernel mailing list -- kernel@lists.fedoraproject.org > To unsubscribe send an email to kernel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson wrote: > So, there was recently a Thing where btrfs installs were broken, and > this got accepted as a release blocker: > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments). How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug? 7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different? I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes? -- Chris Murphy ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Re: Discussion: what would not blocking on btrfs look like?
On 8/23/19 1:10 PM, Neal Gompa wrote: On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes wrote: On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson wrote: Hey folks! So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker: https://bugzilla.redhat.com/show_bug.cgi?id=1733388 The bug was fixed, so that's fine, but along the way, Laura said this: "I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream." and Justin followed up with: "Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking." So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more! This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from: "When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to: * Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions * Create mount points backed by ext4 partitions, LVM volumes or btrfs volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..." so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this: "The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration." with a somewhat apologetic explanatory footnote: "Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..." so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy: "The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs." but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers. All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases. Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects. The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though. I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes wrote: > > On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson > wrote: > > > > Hey folks! > > > > So, there was recently a Thing where btrfs installs were broken, and > > this got accepted as a release blocker: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > The bug was fixed, so that's fine, but along the way, Laura said this: > > > > "I'm strongly against anything with btrfs being a blocker. If that's in > > the criteria I think we should see about removing btrfs simply because > > we don't have the resources to actually deal with btrfs besides > > reporting bugs upstream." > > > > and Justin followed up with: > > > > "Agreed, btrfs has been a gamble pretty much always. See previous > > discussion around proposals to make btrfs default. Ext4 and xfs should > > be the only release blocking." > > > > So, that's the whole kernel team 'strongly against' blocking on btrfs. > > Which means we should talk about not doing that any more! > > > > This is a bit complicated, though, because of how the Final criteria > > are phrased. Basic does not include btrfs at all, and Beta includes a > > laundry list we can just remove btrfs from: > > > > "When using both the installer-native and the blivet-gui-based custom > > partitioning flow, the installer must be able to: > > > > * Correctly interpret, and modify as described below, any disk with a > > valid ms-dos or gpt disk label and partition table containing ext4 > > partitions, LVM and/or btrfs volumes, and/or software RAID arrays at > > RAID levels 0, 1 and 5 containing ext4 partitions > > * Create mount points backed by ext4 partitions, LVM volumes or btrfs > > volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing > > ext4 partitions > > ..." > > > > so those two are easy. However, the Final criterion is not laundry > > list-style. The relevant Final criterion is this: > > > > "The installer must be able to create and install to any workable > > partition layout using any file system and/or container format > > combination offered in a default installer configuration." > > > > with a somewhat apologetic explanatory footnote: > > > > "Wait, what? > > Yeah, we know. This is a huge catch-all criterion and it's subject to a > > lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is > > that you should be able to do anything sane that the Installation > > Destination spoke attempts to let you do, without the installer > > exploding or failing. We are trying to write more specific criteria > > covering this area, but it's not easy. Patches welcome, as the kids > > say..." > > > > so as the footnote says, the rule is basically "if the installer lets > > you do it, it ought to work". It seems a bit awkward to craft an > > exception for btrfs from that. I mean, technically it's easy: > > > > "The installer must be able to create and install to any workable > > partition layout using any file system and/or container format > > combination offered in a default installer configuration, except > > btrfs." > > > > but that's odd. Why is btrfs, alone, an exception? It kinda goes > > against the fundamental idea of the criterion: that we stand behind > > everything the UI offers. > > All of this, the criteria, and the UI support for btrfs are from the > many years old proposal to make btrfs the default filesystem. In the > beginning, it was not ready, but did show promise. This proposal came > up for several releases in a row, and at the end of it, even the > upstream developers recommended against it. At this point, it is safe > to say that btrfs will not be the default. Since that time, things > have not gotten better. Yes, there is active btrfs development > upstream. It is fairly narrowly focused, and not something we can > rely upon for a supported default among the Fedora use cases. Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects. The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though. I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one h
Re: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson wrote: > > Hey folks! > > So, there was recently a Thing where btrfs installs were broken, and > this got accepted as a release blocker: > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > The bug was fixed, so that's fine, but along the way, Laura said this: > > "I'm strongly against anything with btrfs being a blocker. If that's in > the criteria I think we should see about removing btrfs simply because > we don't have the resources to actually deal with btrfs besides > reporting bugs upstream." > > and Justin followed up with: > > "Agreed, btrfs has been a gamble pretty much always. See previous > discussion around proposals to make btrfs default. Ext4 and xfs should > be the only release blocking." > > So, that's the whole kernel team 'strongly against' blocking on btrfs. > Which means we should talk about not doing that any more! > > This is a bit complicated, though, because of how the Final criteria > are phrased. Basic does not include btrfs at all, and Beta includes a > laundry list we can just remove btrfs from: > > "When using both the installer-native and the blivet-gui-based custom > partitioning flow, the installer must be able to: > > * Correctly interpret, and modify as described below, any disk with a > valid ms-dos or gpt disk label and partition table containing ext4 > partitions, LVM and/or btrfs volumes, and/or software RAID arrays at > RAID levels 0, 1 and 5 containing ext4 partitions > * Create mount points backed by ext4 partitions, LVM volumes or btrfs > volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing > ext4 partitions > ..." > > so those two are easy. However, the Final criterion is not laundry > list-style. The relevant Final criterion is this: > > "The installer must be able to create and install to any workable > partition layout using any file system and/or container format > combination offered in a default installer configuration." > > with a somewhat apologetic explanatory footnote: > > "Wait, what? > Yeah, we know. This is a huge catch-all criterion and it's subject to a > lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is > that you should be able to do anything sane that the Installation > Destination spoke attempts to let you do, without the installer > exploding or failing. We are trying to write more specific criteria > covering this area, but it's not easy. Patches welcome, as the kids > say..." > > so as the footnote says, the rule is basically "if the installer lets > you do it, it ought to work". It seems a bit awkward to craft an > exception for btrfs from that. I mean, technically it's easy: > > "The installer must be able to create and install to any workable > partition layout using any file system and/or container format > combination offered in a default installer configuration, except > btrfs." > > but that's odd. Why is btrfs, alone, an exception? It kinda goes > against the fundamental idea of the criterion: that we stand behind > everything the UI offers. All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases. While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention. > So...what should we do? Here are the options as I see 'em: > > 1. Keep supporting btrfs > 2. Just modify the criterion with a btrfs exception, even if it's weird > 3. Rewrite the criterion entirely > 4. Keep btrfs support in the installer (and blivet-gui) but hide it as > we used to - require a special boot argument for it to be visible > 5. Drop btrfs support from the installer > I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases. > I'm bringing this to anaconda, kernel and test teams initially to kick > around; if we agree on an approach we should then probably loop in > devel@ for wider review, unless the choice is 1). > > Thanks for any thoughts, folks! > -- > Adam Williamson > Fedora Q
