On Tue, Nov 29, 2022 at 09:31:01PM +0100, Kevin Kofler via devel wrote: > Hi, > > Stephen Gallagher wrote: > > Following is the list of topics that will be discussed in the > > FESCo meeting Tuesday at 17:00UTC in #fedora-meeting on > > irc.libera.chat. > > does anyone have a complete log? It does not show up on > meetbot.fedoraproject.org, and the raw log at > https://meetbot-raw.fedoraproject.org/fedora-meeting/2022-11-29/fesco.2022-11-29-17.00.log.txt > is truncated.
--- Log opened Tue Nov 29 17:07:07 2022 17:07 -!- zbyszek [~zbyszek@fedora/zbyszek] has joined #fedora-meeting 17:07 -!- Irssi: #fedora-meeting: Total of 183 nicks [1 ops, 0 halfops, 62 voices, 120 normal] 17:07 -!- Irssi: Join to #fedora-meeting was synced in 1 secs 17:07 < zbyszek> .hello2 17:07 < zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbys...@in.waw.pl> 17:07 < zbyszek> Sorry, another meeting refusing ETERM signalling. 17:08 < nirik> ok, back sorry. 17:10 <+sgallagh> #topic #2817 Change proposal: Add -fno-omit-frame-pointer to default compilation flags 17:10 <+sgallagh> .fesco 2817 17:10 -!- zodbot changed the topic of #fedora-meeting to: #2817 Change proposal: Add -fno-omit-frame-pointer to default compilation flags (Meeting topic: FESCO (2022-11-29)) 17:10 < zodbot> sgallagh: Issue #2817: Change proposal: Add -fno-omit-frame-pointer to default compilation flags - fesco - Pagure.io - https://pagure.io/fesco/issue/2817 17:10 < mhroncok> I'm back, thank you 17:10 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 17:11 < decathorpe> hey o/ sorry for being late 17:11 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 17:11 -!- zodbot [~supybot@fedora/bot/zodbot] has quit [Read error: Connection reset by peer] 17:11 -!- jmracek57 [~jmracek@185.16.81.139] has quit [Quit: Client closed] 17:11 < decathorpe> or, reading backlog, not late after all :) 17:12 < mhroncok> check out the latest comment 17:12 < nirik> so... lots of discussion on this one. 17:12 < nirik> right, I was going to ask if the tools folks had really weighed in. 17:12 <+DaanDeMeyer[m]> Hi everyone, I'm joining as well for the frame pointer proposal 17:12 < dcantrell> I was already leaning towards voting no on this one, but if the tools team advises against it as well, that's a solid no from me 17:13 < codonell> :-) 17:13 < dcantrell> I saw codonell join as well 17:13 < nirik> I was trying to think of a middle ground type thing, but it would need a second mass rebuild, which I am not sure would fly. 17:13 < decathorpe> yeah :( if there's objections that strong against it from the people who maintain the affected tools ... 17:13 <+sgallagh> Let's give daandemeyer[m] a minute to state their case, but I think we should make a final decision today in this meeting 17:14 < zbyszek> So, what is the basis for opposition from tools folks? 17:14 < Eighth_Doctor> on the other hand, the tools team might not be solely in the right here, given that performance teams are saying they need this 17:14 < Eighth_Doctor> so there's clearly a big disconnect going on 17:14 <+music[m]> fweimer: Thanks for the update from the platform tools team (https://pagure.io/fesco/issue/2817#comment-829918). That was one thing on which I had meant to request an update, but I hadn't had a chance. 17:15 < Eighth_Doctor> my beef with this isn't with actually doing the change, it's that I didn't know whether people would use the new data to actually do perf stuff 17:15 < Eighth_Doctor> at least from the desktop developers I've talked to, they consider this change highly valuable 17:15 < fweimer> zbyszek: It's substantial slowdown across the board. And there's no interest in frame pointers downstream. 17:15 < Eighth_Doctor> both on GNOME and KDE Plasma 17:15 < fweimer> Not from partners nor customers. 17:15 < fweimer> Or our internal performance team. 17:15 < zbyszek> Eighth_Doctor: I talked to some people privately who use perf for various things, and they were rather enthusiastic about this. 17:16 < Eighth_Doctor> on the other hand, I also wonder why nobody has tried to make -O3 better either 17:16 < zbyszek> fweimer: So those are very weak arguments, sorry. 17:16 < nirik> did I read somewhere that ppcl4le/aarch64 do have frame-pointers by default? 17:16 <+DaanDeMeyer[m]> I can only say that we run upstream glibc compiled with frame pointers internally and haven't run into any noticeable issues, we still get tremendous value out of building with frame pointers 17:16 < fweimer> nirik: Correct. 17:17 < Eighth_Doctor> the other think I've heard from some folks is that the reason we have this problem is similar to why we had this when spectre/meltdown mitigations were first implemented 17:17 < codonell> DaanDeMeyer[m], Do you have patches you can contribute upstream to fix all the missing frame pointer in the assembly code? :-) 17:17 < nirik> would perf work done on them translate to x86_64? 17:17 < Eighth_Doctor> nobody bothered to do performance with frame pointers for so long that the only real way to make it back is to actually start doing it again 17:17 < fweimer> zbyszek: Why is this a weak argument? I think we're fairly well-connected. 17:17 < Eighth_Doctor> s/think/thing/ 17:18 < zbyszek> fweimer: re "slowdown": the slowdowns are not that big and the expectation is that most of them will be work-around as this is implemented. 17:18 < Eighth_Doctor> fweimer: because partners and customers don't know what any of it means 17:18 < zbyszek> re "no interest": there is clearly interest from people who care and use this. That people who don't use it and don't care don't care, doesn't mean much. 17:18 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has joined #fedora-meeting 17:18 -!- mode/#fedora-meeting [+v anitazha] by ChanServ 17:19 < fweimer> Eighth_Doctor: That's extremely condescending. I assume they have learnt how to work with the tools we provide. 17:19 < Eighth_Doctor> and we're going to take the hit for Python anyway in 3.12 most likely 17:19 < Eighth_Doctor> which is actually where the biggest pain is 17:19 <+davide> .hello dcavalca 17:19 < zbyszek> Eighth_Doctor: … python upstream will work on making the pain go away. 17:19 < codonell> zbyszek, Eighth_Doctor, The slowdowns are real. There is real register pressure on x86_64, that is different from other architectures that have different ISA designs. 17:19 < Eighth_Doctor> fweimer: it's borne from experience... this stuff is weirdly deep and arcane 17:20 < michel-slm> .hi 17:20 < Eighth_Doctor> and a lot of it is cargo culting 17:20 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has left #fedora-meeting [] 17:20 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has joined #fedora-meeting 17:20 -!- mode/#fedora-meeting [+v anitazha] by ChanServ 17:20 <+DaanDeMeyer[m]> codonell: We don't have any glibc patches for fixing the assembly frame pointers issues, we run upstream glibc as far as I'm aware 17:20 < codonell> Eighth_Doctor, What is cargo culting? That we should not enable frame pointers? 17:20 < decathorpe> Would it be possible to do something similar to what eln does with rawhide? I.e. rebuild (a relevant subset of) rawhide in rawhide-fp? That would let people opt-in to these packages if they need to. And when / if the worst problems have been mitigated, it could be reconsidered for a default change? 17:20 < Eighth_Doctor> Fabio Valentini: it has to be the whole distro 17:20 < Eighth_Doctor> otherwise it's not useful for developers 17:20 <+davide> yeah, a subset isn't actually useful 17:20 < Eighth_Doctor> especially desktop devs 17:21 < michel-slm> if the register pressure is heaviest on x86_64, there's also the possibility of enabling it for other arches first, right 17:21 < Eighth_Doctor> codonell: -O2 vs -O3, no frame pointers, other compilation flags we have 17:21 <+sgallagh> Fabio Valentini: FWIW, if we wanted to add a new target to Koji for this, the tools I wrote for ELN would work just fine for those builds. 17:21 < Eighth_Doctor> most people have no clue what that even means and lots of options are used or unused based on lore 17:21 < decathorpe> Davide Cavalca: I think you missed the "relevant" specifier for "subset" ... 17:21 < fweimer> michel-slm: It's already enabled for aarch64 & ppc64le. 17:22 < Eighth_Doctor> michel-slm: it's already on for aarch64 and power 17:22 <+sgallagh> michel-slm: From what was said above, the other arches already have frame pointers 17:22 < Eighth_Doctor> the missing ones are x86 and s390x 17:22 < decathorpe> Stephen Gallagher: right, that's why I thought that this might be a way to salvage all the work that went into this proposal 17:22 < michel-slm> I wonder what the consideration is for s390x 17:22 < nirik> right, thus my question... could we just ask folks to use those arches for perf work? or does the kind of things they are doing not really translate? 17:22 <+davide> Fabio Valentini: that's my point, "relevant" in this context is the whole distro, due to the dependency web for any non-trivial application 17:22 <+davide> this is especially true for continuous profiling applications 17:23 < decathorpe> I meant: excluding stuff like noarch packages and ocaml which doesn't support frame pointers ... 17:23 < zbyszek> nirik: you can't really do profiling of desktop stuff and such… Access to ppc64el and arm64 machines is hard for most poeple. 17:23 <+davide> ah got it, thanks for clarifying 17:23 < nirik> rpi4 seems like it would be pretty doable, but sure... 17:23 < codonell> Eighth_Doctor, They aren't "missing", the x86 and s390x choices are specific. 17:24 < Eighth_Doctor> rpi4 is useless for profiling for other reasons 17:24 <+music[m]> Yeah, if I'm J. Random Developer, I'm definitely not going to go buy an exotic workstation for profiling. And rpi4 is going to have different bottlenecks from a desktop-class system. 17:24 < Eighth_Doctor> it's a horrible arm64 platform 17:24 <+music[m]> Much less a server-class system. 17:24 < nirik> music[m]: would you install a fedora remix? 17:24 <+davide> profiling work is pretty system specific, I don't think it's realistic to expect folks to keep a rpi on their desk for this, and the results wouldn't be comparable anyway 17:24 < Eighth_Doctor> codonell: yeah, if that's true, they're not documented 17:25 < Eighth_Doctor> I literally only was able to learn about why we have the omit frame pointer thing from Brendan Gregg 17:25 < Eighth_Doctor> nobody has docs anywhere about it 17:25 <+DaanDeMeyer[m]> Even a remix wouldn't work, because it's a different system, you can't even ask a bug reporter to do a profile for you and provide the results because they'd have to switch to the remix 17:25 < codonell> Eighth_Doctor, Would you like me to document that somewhere? 17:26 < Eighth_Doctor> yes, please 17:26 < Eighth_Doctor> document all of our compilation options and their rationales 17:26 <+music[m]> nirik: if a remix existed, i think it would be of some use, but very few people would download and install in a VM to do profiling in practice 17:26 < Eighth_Doctor> I only know the cargo-cult-y reason why we've never done -O3, not the real reason why we never tried, for example 17:27 <+music[m]> i think that it's likely a remix it would not be maintained long-term in practice 17:27 < nirik> yep, its a lot of work 17:27 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 17:27 < dcantrell> looking at other distributions and platforms, profiling builds of everything have usually been offered alongside the "regular" builds. the general understanding being they are slower but they are also available for those interested 17:27 < Eighth_Doctor> also there's legal problems with remixes 17:27 < Eighth_Doctor> in that the whole trademark policy is a mess and nobody knows how to deal with it right now 17:27 < fweimer> codonell: A while back, I documented all the Fedora-specific choices (buildflags.md). But this one is not Fedora-specific, it comes straight from upstream. 17:27 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 17:27 <+davide> a remix also doesn't help you if you're currently running Fedora, have an issue in production and want to use profiling to troubleshoot 17:27 <+music[m]> and as daandemeyer pointed out, you're either installing the remix on different hardware or on a vm; in either case it isn't the original system of interest 17:28 < mhroncok> brainstorming: would it make sense to enable this in rawhide only and evaluate the results in time for a quick revert and rebuild before F38 beta? 17:28 <+davide> it's not realistic to switch a system on the fly to the remix, and a separate system with the remix wouldn't necessarily help in troubleshooting (or provide the same results) 17:28 < dcantrell> yeah, not really in favor of that 17:28 < fweimer> Eighth_Doctor: -O3 is not generally faster than -O2. GCC 12 did enable auto-vectorization at -O2 though, with a cost model that is beneficial. 17:28 < mhroncok> (rather than having it as an experiment for 1 year which indeed seems rather long) 17:28 < nirik> mhroncok: would need a mass rebuild... 17:28 < fweimer> So there is movement. 17:28 < Eighth_Doctor> we get observability principle problems when you do that 17:28 < michel-slm> mhroncok: you'll need an extra mass rebuild, yeah 17:28 < codonell> Eighth_Doctor, The knowledge is largely maintained by downstream and IHV toolchain teams, and the CPU design teams that support them. We are here to support today's decision with that knowledge and experience. 17:29 < gotmax> Well, it would be possible to distro-sync a regular rawhide system to the theoretical rawhide-fp repos w/o a reinstall. 17:29 < zbyszek> gotmax: you'd still likely get different versions of packages. 17:30 < Eighth_Doctor> one outcome I'd prefer to see is figuring out how to make -fno-omit-frame-pointers more performant if that's the core issue 17:30 <+music[m]> agree with fweimer re: -03 and Davide Cavalca re: remixes 17:30 < zbyszek> Also… not everybody runs rawhide everywhere, so we're back to the square that we want to do this on production systems and not a fresh system. 17:30 <+sgallagh> Note: we are at 20 minutes on this topic. 17:30 < zbyszek> So… considering that we need a mass rebuild to really see the effects, and we are worried that some specific applications might be negatively impacted, 17:30 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 17:30 < Eighth_Doctor> like register pressure is a real thing, I get it, but at the same time, having Linux be worse than Windows and macOS is bad too 17:30 <+sgallagh> Does anyone want to make a proposal more detailed than "accept or reject the Change"? 17:30 < gotmax> zbyszek: Fair point about not running rawhide, but automatic rebuilds should keep versions in sync if they're done properly. 17:31 < mhroncok> can we do some preliminary vote to see if the discussion even makes sense? 17:31 < fweimer> Eighth_Doctor: There's also the option to find a way to create useful flamegraphs *without* frame pointers. 17:31 < Eighth_Doctor> gotmax: with what tooling? 17:31 < fweimer> I don't understand why that's not really on the table. 17:31 < gotmax> There's the ELN tooling that sgallagh brought up 17:31 <+sgallagh> Conan Kudo: presumably they'd use DistroBuildSync like ELN does 17:32 < Eighth_Doctor> fweimer: does anyone care in the toolchain team to enable that? my understanding from gnome developers is that they've been brushed off on this for over a decade 17:32 < nirik> fweimer: is anyone actually working on that? 17:32 < Eighth_Doctor> so unless there's a change in attitude there, then 🤷♂️ 17:32 < Eighth_Doctor> and there's also problems at the kernel level, since no dwarf unwinding is ever going to happen 17:33 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 17:33 < Eighth_Doctor> Stephen Gallagher: it doesn't do build chains properly, though, I see that fail a lot 17:33 <+sgallagh> That's a tangent and can be discussed IF we end up taking that approach 17:33 < codonell> Eighth_Doctor, The kernel has specific tradeoffs that make it choose Orc or CTF. Those formats are growing in size every year, approaching Dwarf in complexity as they attempt feature parity :-) 17:34 <+sgallagh> Last change to make a more detailed proposal, else I'm holding the Accept or Reject vote in 60s 17:34 < nirik> FWIW, i'm -1 to the change as is, open to revisiting if python is no longer so affected/benchmarks show its not as bad. 17:34 < zbyszek> proposal: enable this for F38 and watch out for performance regressions. If there are big performance regressions that can't be resolved, we can quickly rebuild specific packages with -fomit-frame-pointer, and if the overall outlook is bad, revert it altogether. 17:35 < Eighth_Doctor> codonell: yeah, but until they do dwarf, we're stuck 17:35 < nirik> zbyszek: by 'enable' you just mean for things that rebuild? 17:35 -!- besser82[m] [besser82@fedora/besser82] has quit [Quit: Freedom, Friends, Features, First ---> https://getfedora.org] 17:35 < zbyszek> By "enable" I mean make the change. 17:35 < codonell> Eighth_Doctor, I have no objection to setting up a collaboration with GNOME if they need support form toolchain people to implement what they need. 17:35 < gotmax> zbyszek: Here we come back to the people don't run rawhide problem. Issues will likely come up later in the release cycle when it's kind of late to revert things. 17:35 < zbyszek> Flip the switch. 17:35 < mhroncok> zbyszek: does your proposal consider that Python 3.11 would opt-out? 17:35 <+sgallagh> zbyszek: Or do you mean mass rebuild 17:35 < fweimer> Eighth_Doctor: We could work with GNOME. We've put in a lot of effort to get fast backtraces going over the years. 17:35 < zbyszek> OK, let me rephrase. 17:35 < mhroncok> and, who "watches out for performance regressions"? 17:36 < fweimer> It's just that it's not about frame pointers, so it gets dismissed instanatly by many. 17:36 < Eighth_Doctor> fweimer: and KDE? because they have similar issues 17:36 < codonell> Eighth_Doctor, "Brushed off" is certainly not how we want to engage with users of our tooling. 17:36 <+DaanDeMeyer[m]> zbyszek: As mentioned in the fesco ticket, we're in favor of this approach 17:36 <+davide> the Change owners would be happy to help investigate performance regressions that arise from this 17:37 <+music[m]> as mhroncok said, the hard part is, who is going to do the work to find the regressions before release 17:37 < zbyszek> proposal: Flip the switch as proposed for F38 and watch out for performance regressions. If there are big performance regressions reported that can't be resolved, managers can opt-out for specific packages and rebuild with -fomit-frame-pointer, and if the overall outlook is bad, we can consider reverting the change altogether. 17:37 <+davide> and we had already started doing so for things like Python that had already been surfaced 17:37 < fweimer> Eighth_Doctor: Well, the plan is to enable this in the perf APIs, so it shouldn't matter much which frontend you use. 17:37 < decathorpe> problem is that most things will only apply the change after mass rebuild, and at that point we'd need a second mass rebuild to revert it. 17:37 * nirik agrees with decathorpe 17:38 < zbyszek> decathorpe: yeah, but we don't need to rebuild everything. It's most likely to be some small subset of packages that have problems. Maybe python, maybe glibc. 17:38 <+davide> when is the mass rebuild planned for? 17:38 < dcantrell> I feel we do way too many mass rebuilds as it is 17:38 < Eighth_Doctor> fweimer: in practice, that's not fully true 17:38 <+sgallagh> zbyszek: "Flip the switch" isn't sufficiently clear. Do you mean "change the macro and let things rebuild when they happen'" or do you mean "run a mass-rebuild to ensure everything is built with this flag" 17:38 < zbyszek> The former. 17:38 < Eighth_Doctor> dcantrell: we should be able to do so many that they're a nonevent 17:38 < Eighth_Doctor> the reason they're hard for us is that our tooling makes it hard 17:38 < dcantrell> Eighth_Doctor: I disagree 17:38 <+sgallagh> zbyszek: -1 17:38 < fweimer> sgallagh: Flip the switch is clear, but it doesn't mean that you get frame pointers. 17:38 < nirik> Wed 2023-01-18 is mass rebuild for f38 17:39 < fweimer> sgallagh: So it's not clear to me whether the proposal is about getting frame pointers, or just some textual change to redhat-rpm-config. 17:39 * nirik also notes few people will be looking at this over the holidays either 17:39 < dcantrell> nirik: a very valid point 17:39 < gotmax> I don't think changes that we plan to revert if things go wrong are helpful. If there's not confidence in them, they probably shouldn't be approved. 17:39 < dcantrell> gotmax: I agree 17:40 <+sgallagh> We're now at 30 minutes. Let's at least see where votes fall on a strict "yes or no" vote on the Change. 17:40 <+sgallagh> I'm -1 at this time 17:40 < dcantrell> -1 17:40 < Eighth_Doctor> +1 17:40 < zbyszek> +1 17:40 < gotmax> People were unhappy about this part of the SHA-1 change. I don't see why this should be treated differently. 17:40 < decathorpe> 0 17:41 < nirik> -1 17:41 < Eighth_Doctor> I don't find the toolchain team's arguments compelling without a credible alternative from them 17:41 < mhroncok> -1 17:41 < Eighth_Doctor> and so far, we have nothing, and a lot of suffering developers who can't do perf work 17:42 <+sgallagh> I count (+2, 1, -4) so far. 17:42 <+sgallagh> mhayden ? 17:42 < Eighth_Doctor> in many respects, this is more useful than annobin, package notes, and all the other weird changes we've made to the compiler stack 17:42 < Eighth_Doctor> and both of those examples have regularly broken the toolchain 17:43 <+sgallagh> (Note that a 0 vote is functionally a -1 here, since it doesn't result in a change) 17:43 < zbyszek> Eighth_Doctor: yeah, this has the potential to be quite useful, and we won't actually see this potential until it's been available for a while and people learn to use it more widely. 17:43 -!- daandemeyer [~daandemey@2a02:a03f:864b:8201:e534:34f4:1c34:8de7] has joined #fedora-meeting 17:43 <+sgallagh> It's not possible for this to pass, given the votes it has received. 17:44 < fweimer> We'll raise it with our IHVs nevertheless. 17:44 < Eighth_Doctor> laying the cards on the table: I don't even like this change because I don't like taking a perf hit 17:44 < Eighth_Doctor> but what I like even less is how badly desktop developers get screwed for development tooling 17:45 < Eighth_Doctor> I know Meta wants this for perfing their workloads, but I want it to make desktop developers' lives better 17:45 < mhroncok> fweimer: What does IHVs mean? 17:45 < michel-slm> indep hardware vendor? 17:45 < Eighth_Doctor> mhroncok: Independent Hardware Vendors 17:45 < nirik> Hopefully we get some better options for them... 17:45 < fweimer> mhroncok: Independent Hardware Vendors (silicon in this case) 17:45 <+sgallagh> mhayden, music: Last chance to vote 17:46 < mhroncok> cool, I was confused because I found the Institute of Human Virology 17:46 < Eighth_Doctor> nirik: I'm honestly not optimistic 17:46 < mhroncok> is it important to get this now? 17:46 < Eighth_Doctor> desktop Linux investment is lower than it has ever been overall 17:46 < mhroncok> why don't we wait for the Python 3.12 situation to unfold 17:46 <+sgallagh> #agreed REJECTED - Add -fno-omit-frame-pointer to default compilation flags (+2, 1, -4) 17:46 < mhroncok> and reconsider this change then? 17:46 < Eighth_Doctor> I just don't see anyone caring enough to add new tools to make their lives better 17:47 <+sgallagh> A new proposal can be submitted at a later date if new information comes to light 17:47 < mhayden> sorry y'all, got a new puppy and had a bathroom emergency 🤦♂️ 17:47 < Eighth_Doctor> if I were skilled in such a way, I'd do something about it :( 17:47 < michel-slm> mhayden: aww, congrats 17:47 < Eighth_Doctor> but I'm not 17:47 <+sgallagh> #2870 Change proposal: Replace DNF with DNF5 17:47 <+sgallagh> .fesco 2870 17:48 * decathorpe waits for music to finish typing 17:48 <+music[m]> I didn't get my vote in quickly enough, but I'm going to go on record as weakly in favor (despite compelling arguments from both positions), for similar reasons to Conan Kudo . 17:48 <+sgallagh> #info music votes +1, belatedly 17:48 <+sgallagh> I'll include that in the tally on the ticket 17:48 < Eighth_Doctor> it'd be almost a tie if mhayden voted in favor :P 17:48 <+sgallagh> Let's move on, as we're already approaching the top of the hour 17:49 < mhroncok> dnf -- I more or less agree with what Stephen Gallagher commented recently in the ticket 17:49 < gotmax> It seems zodbot didn't respond to the .fesco command... 17:49 <+sgallagh> .fesco 2870 17:49 < nirik> on this one, theres some dispute about the replacement? 17:50 * dcantrell is catching up on the ticket 17:50 < Eighth_Doctor> meh, sure I'll go with what Stephen Gallagher said 17:50 < Eighth_Doctor> I don't know if I want machine-readable output from dnf though 17:50 < Eighth_Doctor> that would encourage bad things from people 17:50 < Eighth_Doctor> I'd rather they use the API and build things around it 17:50 < decathorpe> problem: the API is horrible, CLI is not as bad 17:50 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 17:50 < Eighth_Doctor> the only reason zypper has machine readable output is because zypper's API is absolute crap 17:51 < mhroncok> OTOH I am afraid that the change owners are not committed to "to at least enumerate the major pieces that will need to be modified (infra and releng in particular) as part of the Change page" 17:51 <+sgallagh> Let's not go into the weeds on that request. 17:51 < Eighth_Doctor> Stephen Gallagher: you put it in there :P 17:51 <+sgallagh> jmracek: Are you around? 17:51 < decathorpe> mhroncok: I agree, detemining change impact is not job for the community 17:51 < gotmax> They have updated the scope section a bit 17:51 <+sgallagh> I did, but not as something conditional for approval 17:51 < jmracek> Yes 17:52 <+sgallagh> That may have been unclear 17:53 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 17:53 -!- daandemeyer [~daandemey@2a02:a03f:864b:8201:e534:34f4:1c34:8de7] has quit [Quit: Client closed] 17:54 <+sgallagh> jmracek: If you and your team could do the legwork to at least identify the major infra and releng pieces that might be broken and need to be kept in mind, I think that would be sufficient for me to vote +1 here. 17:54 * nirik is in favor of the idea of this change, but not sure about some of the details/etc 17:54 <+sgallagh> Could you commit to that? 17:54 < jmracek> I tried my best to answer all the questions in all chanels and I improved proposal according to suggestions. Hope I succeed 17:54 < decathorpe> "Repoquery command with unknown scope" lists that repoquery options that are essential for Fedora development are still missing, things like "--whatrequires" and "--requires" ... that doesn't really make me confident 17:54 < Eighth_Doctor> I would not vote until we have that list enumerated to evaluate 17:54 < Eighth_Doctor> cart before the horse after all 17:55 <+music[m]> decathorpe: Where are you reading this from? 17:55 < gotmax> https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5#Scope 17:55 < decathorpe> music: https://github.com/rpm-software-management/dnf5/issues/122 17:56 < jmracek> python3-dnf will be arrond, dnf-3 binary, therefore minimal list is quite short 17:56 < decathorpe> the list of things that are still missing compared to dnf4 looks rather bleak right now 17:56 < jmracek> The list of all dependencies is in proposal 17:56 -!- anakryiko [~anakryiko@163.114.132.128] has quit [Quit: Client closed] 17:56 < dcantrell> the proposal presents a lot of the work items and lacks a lot of "how this will look to the end user" details. that's what I would like to see. emphasis on minimizing impact and not breaking things, obviously 17:57 < gotmax> jmracek: That's true, but if dnf5 is the default, "dnf-3 still exists" is not a solution to "there's a lot of missing functionality" 17:57 < decathorpe> yeah,dependencies are listed, but not dependents 17:57 < jmracek> We got a minimal responce from comunity what they need 17:58 <+sgallagh> Yes they are 17:58 < jmracek> We have some time to delivery missing functionality 17:59 <+sgallagh> My biggest concerns are the pungi, lorax and mock use-cases. 17:59 < jmracek> Without the approval, no one will care about DNF5 17:59 < gotmax> Perhaps there should be another release in between "dnf5 is available in the repos" and "dnf5 is the default"? 17:59 <+sgallagh> User-interaction is honestly low on my list; users are more adaptable 17:59 <+music[m]> Maybe I misunderstand that statement, but I'm not sure that keeping existing features should be considered "opt-in" based on feedback from the relatively small group of people that are paying attention to dnf5 at this point. 17:59 < decathorpe> The way I see it, voting on making dnf5 take over from dnf before it can make essential functionality work is way premature 18:00 <+sgallagh> gotmax: Reading between the lines, I assume that DNF5 is wanted for RHEL 10, which is expected to branch from Fedora 40, IIRC 18:00 < Eighth_Doctor> I'm also not super pleased about them ripping out the abbreviations from dnf5 18:00 < gotmax> Eighth_Doctor: You mean the zypper style ones? 18:00 < Eighth_Doctor> yes 18:00 <+sgallagh> And I for one would not want RHEL 10 to ship without at least one full Fedora release to bake it first 18:01 < jmracek> I don't say that the change will be not painfull, but this is not the first time. Right now only tools that use commandline will be directly affected, but DNF5 will be compatible in many aspects 18:01 < gotmax> sgallagh: Sure, but Fedora makes decisions based on its interests first and foremost. 18:01 < nirik> this is for f39 right? 18:01 < Eighth_Doctor> Stephen Gallagher: yes... that's been a point in the last couple dnf community meetings 18:01 <+music[m]> I feel like I don't have a good idea of what kind of functionality dnf5 will and won't have by the time it would take over as default. 18:02 < nirik> music[m]: yeah. 18:02 < Eighth_Doctor> nirik: yes 18:02 <+sgallagh> gotmax: I try not to make decisions in a vacuum. (Particularly since RHEL is my $DAYJOB). 18:02 < gotmax> <jmracek> Without the approval, no one will care about DNF5 | I don't think that's necessarily true 18:02 < jmracek> mock can already test and we are in contact 18:03 < jmracek> If I am not mistaken pungi, lorax uses python3-dnf (dnf api) therefore not effected 18:03 < nirik> in releng land: releng scripts call dnf, pungi needs to work, koji of course needs to work (meaning mock, etc too), the weird image and container stuff we have 18:04 < Eighth_Doctor> kiwi needs to work for fedora asahi and hyperscale stuff 18:04 < Eighth_Doctor> but that I expect will be fine 18:04 < Eighth_Doctor> since kiwi and dnf have been working together for a while 18:04 <+music[m]> can we clarify @nirik's question about F39 vs F38? there was discussion in the issue about doing this for F39, but the Change page is for F38 18:04 < decathorpe> service which provides broken dependencies report for the packager dashboard uses dnf repoclosure, which seems to be gone entirely 18:04 < jmracek> You can agree and we can set that those tools must work at certain time point. Is it fair for you? 18:04 <+sgallagh> I'm also somewhat concerned that FESCo members are falling prey to the "nirvana fallacy" 18:04 <+sgallagh> music: The Change page says F39 18:05 < Eighth_Doctor> well, I have to go now... dental appointment awaits 18:05 < Eighth_Doctor> bye y'all 18:05 < dcantrell> take care 18:05 < nirik> jmracek: it would definitely be good to try and get things all working before f38 cycle... if possible. 18:05 < nirik> f39 I mean 18:05 < nirik> Eighth_Doctor: ouch. Good luck. 18:05 <+music[m]> Ah, I was in the wrong place. Got my tabs straightened out. Thanks. 18:05 < dcantrell> the previously mentioned suggestion of landing dnf5 parallel for one release and then have a separate change to switch over to it as the default would make me feel better 18:06 < decathorpe> but aren't we back to "we shouldn't enable it if if we're not confident that it can work", aren't we? 18:06 < dcantrell> switch over to it in the following release, that is 18:06 < gotmax> dcantrell: Isn't that what we already have? 18:06 <+sgallagh> There's no need to have a Change for F38 to land it 18:06 < zbyszek> dcantrell: that's already happening. 18:06 < mhroncok> I am sorry but in my worldview if somebody proposes to change things, they evaluate the impact and propose a solution to the negative impact. 18:06 < zbyszek> dcantrell: it's already in rawhide, i.e. it'll be in F38 if nothing else changes. 18:07 < dcantrell> alright, so this change proposal isn't even that clear to me then 18:07 < decathorpe> yeah, I don't think "let's approve this and wait for people to come screaming because everything is on fire" is a good approach 18:07 < jmracek> In Fedora 38, DNF5 replaces microdnf 18:07 <+sgallagh> dcantrell: What are you missing? 18:07 < dcantrell> decathorpe: agreed 18:07 <+sgallagh> Fabio Valentini: That's different from other approvals... how? 18:07 <+sgallagh> (Sorry) 18:08 < decathorpe> for other Changes, impact and scope are defined much clearer 18:08 < decathorpe> here we're talking about replacing the package manager, which affects everything and everybody 18:08 < gotmax> dcantrell: The change is to obsolete the current dnf and replace the dnf binary with a symlink to dnf5 18:08 <+sgallagh> Well, we're talking about the basic package manager. They could just write "everything" and it wouldn't really be a lie 18:09 < dcantrell> ok, so doing that without existing functionality implemented seems like a pretty easy decision to me. don't make that change until everything we need works 18:09 < decathorpe> true. so there should also be a higher burden to make impact and scope clearer ... 18:10 < nirik> isn't 'everything we need working' what the proposal says? or in not enough detail? I am not sure how easy it is to define 'everything we need working' 18:10 <+sgallagh> dcantrell: And we're back to the Nirvana Fallacy 18:10 < jmracek> I tried to get the list of required functionality from comunity an all requirements were mentioned in proposal. 18:10 < zbyszek> Another thing which is not clear to me: if tools that depend on python3-dnf continue to use that, but other tools use dnf5, how likely are we to get unexpected discrepancies in behaviour? 18:10 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 18:10 < decathorpe> Stephen Gallagher: isn't approaching Nirvana a desirable thing? I don't know that idiom. 18:10 < gotmax> Yeah, currently the contingency plan section is blank. At the very least, the proposal should say that the symlink will only be changed over *after* the listed features are implemented. 18:11 < nirik> jmracek: I am sure we will hit things we didn't know about/notice/realize. ;) 18:11 <+sgallagh> Fabio Valentini: https://en.wikipedia.org/wiki/Nirvana_fallacy 18:11 < dcantrell> I think it can be further grouped by must haves and nice to haves. 18:11 < jmracek> We are developing RFE according to priority and community requirements. 18:11 < decathorpe> jmracek: "I tried to get the list of required functionality from comunity" how did you ask for community feedback? Community Blog post? Fedora Magazine Article? 18:11 < jmracek> nirik: sure and we are ready to face that chalenge 18:12 < gotmax> I started that mailing list thread about dnf5 blockers, because I felt the community's feedback wasn't being collected 18:12 < dcantrell> nirik: I would say that the required functionality to make a release should be a must have, as an example 18:12 < jmracek> We already have 2 fedora-devel topics 18:12 < zbyszek> jmracek: I see "The scope of the features for Fedora 39". This is pretty clear. But there's also long list in "Dependencies". What is the plan for those? 18:13 < jmracek> Upstream have open discussions and issues. 18:13 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 18:13 < jmracek> I had a presentation about DNF5 18:13 < bcotton> The F39 System-Wide Change proposal deadline is 2023-06-27. Could we defer this decision until, say, the F38 Beta release and see where dnf5 stands at that point? 18:13 < jmracek> I think it was Fedora Hatch 18:14 < gotmax> <jmracek> Upstream have open discussions and issues. | That's true, but I don't think that counts as *collecting* 18:14 < decathorpe> jmracek: these are all things with pretty limited / targeted audience ... and probably won't reach the community. 18:14 < jmracek> I am open for any ideas 18:14 < nirik> bcotton: or we could approve it now, but add a 'check in' for then to see how it's going? 18:15 < dcantrell> I am losing my ability to think. It's past lunch time here and I had breakfast at 6am 18:16 < gotmax> jmracek: decathorpe already listed some 18:16 < decathorpe> that would mean we need to know what amount of "brokenness" would be enough to result in a revert to dnf 4, right? 18:16 < jmracek> If we will be unable to deliver what we promise the proposal can be always postponed 18:16 <+sgallagh> Let me rephrase the question: is anyone opposed to including DNF5 as the default package manager in F39, assuming it meets certain TBD requirements? 18:16 < gotmax> Starting a forum thread might also be a good idea 18:17 < decathorpe> Stephen Gallagher: depends on the TBD requirements. that question doesn't make sense to me 18:17 < mhroncok> Stephen Gallagher: such question is potentially useless. 18:17 < zbyszek> sgallagh: would "TBD" be resolved before we make the vote? 18:17 < gotmax> sgallagh: In addition to having the requirements properly enumerated, I'm worried that there won't be enough testing by that point. 18:17 <+sgallagh> The intent of the question is to determine if we're quibbling over the details or the plan as a whole 18:18 < decathorpe> if "TBD requirements == it's perfect", then I'm +1. otherwise, I'm leaning towards the negative 18:18 < decathorpe> :D 18:18 <+sgallagh> If we have general agreement that we want DNF5 to happen and just disagree on specifics of how, that' 18:18 < mhroncok> exactly 18:18 < zbyszek> sgallagh: I think we're quibbling over the *lack* of details. 18:18 <+sgallagh> s very different from "we are opposed to DNF 5" 18:18 -!- fweimer [~fwei...@albireo.enyo.de] has quit [Quit: .] 18:19 < zbyszek> I don't want to agree to a switch when we can't answer basic questions about what tools we use to build and manage the distro will work. 18:19 < decathorpe> Stephen Gallagher: I don't think that makes much sense. people working on packaging tools clearly want to ship DNF5. if we say "we don't want that" we're going to be stuck with an unmaintained dnf4, aren't we? 18:19 < mhroncok> zbyszek: +1 18:19 < dcantrell> zbyszek: +1 18:19 < gotmax> sgallagh: I don't think the problem is opposition to dnf5, it's compatibility and missing functionality 18:20 < zbyszek> decathorpe: Yeah, I think the switch is inevitable in the long run. But it may be better for Fedora to *delay* the switch if some important things are missing. 18:20 <+sgallagh> You're all demanding extra information, but is there literally any information that would change your ultimate decision about allowing DNF5? 18:20 < jmracek> I guess that this is a requirement - tools we use to build and manage the distro will work 18:20 < dcantrell> sgallagh: a switch to zip files 18:20 <+sgallagh> OK, then the question is not "Should we switch?" but "When should we switch?" 18:20 < mhroncok> I don't think there is anybody on FESCo who would argue that we should avoid eventually switching to dnf5 18:21 < jmracek> DNF team will not release DNF5 by the way that we will break Fedora 18:21 < gotmax> How do you even define breaking Fedora? 18:22 < mhroncok> does "tools we use to build and manage the distro" include tools our packagers use? 18:22 <+sgallagh> mhroncok: So what we're really determining is "what requirements should we set to switch in F39 vs. deferring to F40?" 18:22 < jmracek> Without setting firm plan we can repeate the issue with yum, because it remained unsupported in distro for too long 18:22 < dcantrell> (brb, have to get sandwich) 18:22 < decathorpe> A list of features that are present in dnf but removed from / not implemented in / not planned for DNF5 would be great. I can't find that right now. That would likely be more actionable than "we want to know what you need", i.e. people can just check that list if the things they use are missing from DNF5 18:22 < gotmax> "How do you even define breaking Fedora?" | I feel people have asked this type of question in several different ways, and it hasn't been satisfactorily answered. 18:23 < jmracek> I am sorry, I have to disconnect 18:23 < mhroncok> Stephen Gallagher: I think that is more or less what I've been saying since day 1 18:23 <+sgallagh> So let's provide some minimum requirements "Such as: ODCS must be able to complete a full compose of all blocking media" 18:23 <+sgallagh> mhroncok: Maybe, but I'm trying to make sure we're all speaking the same language. 18:23 < zbyszek> decathorpe: + 18:23 < zbyszek> decathorpe: +1 18:23 < gotmax> decathorpe: Strongly agree 18:23 < mhroncok> sure, np 18:24 < jmracek> The list is already in preparation 18:24 <+sgallagh> Another requirement: "The above compose must be comprised of packages built from Koji using DNF5 for mock" 18:25 <+sgallagh> I don't think we want to set requirements on individual features. I think we need use-cases that they can support. 18:25 < gotmax> Well specific features are needed to support those use cases 18:26 < jmracek> Builddep is already implemented therefore I dont see a problem with mock. But during testing there could appear some issues, but this is normal life 18:26 -!- jmracek [~jmracek@185.16.81.139] has quit [Quit: Konversation terminated!] 18:26 < mhroncok> "The list is already in preparation" -- let's wait for it? I don't think this discussion is very productive at this point 18:26 < dcantrell> (back) 18:26 < gotmax> Yup 18:26 * nirik nods 18:26 < decathorpe> +1 18:27 < gotmax> jmracek has left. Not sure if that shows up on Matrix 18:27 < decathorpe> And collecting requirements is probably also going to be more productive once users know what they're about to lose ;) 18:28 <+sgallagh> jmracek: I'm not asking you to defend the current state. Just trying to give you something to actually work with 18:29 * decathorpe 's brian starts to feel like mush 18:29 < dcantrell> also your brain? 18:29 < mhroncok> Fabio Valentini: unfortunately, mush is not supported in dnf5 :D 18:30 < dcantrell> :) 18:30 < decathorpe> apparently. is there something else we need to discuss today? I'd say postpone dnf5 stuff until at least next week ... 18:30 < zbyszek> decathorpe: +1 18:30 <+sgallagh> Let's also take the FESCo interview questions to next week, since we're well over time 18:31 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has quit [Read error: Connection reset by peer] 18:31 < mhroncok> #info a list of things missing from dnf5 is already in preparation, let's postpone the dnf5 discussion until we see it 18:31 <+sgallagh> +1 18:31 < mhroncok> Stephen Gallagher: we cannot do that 18:31 < mhroncok> I am afraid 18:31 * mhroncok checks 18:31 <+sgallagh> Sorry, I meant "tot he ticket" 18:31 < mhroncok> "In case the selection is not complete by 30 November, I will use the same set of questions as the last election cycle" 18:32 < decathorpe> tag it with "OMG!Urgent!!111" 18:32 < zbyszek> We have another 4.5 h. 18:33 -!- jontyms2707749 [~jont...@76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has joined #fedora-meeting 18:33 * sgallagh sighs 18:33 <+sgallagh> #2895 Election interview questions: FESCo (F37) 18:33 <+sgallagh> .fesco 2895 18:34 < mhroncok> judging by the list of candidates, we can keep the questions as they were, copy our interviews from last year and get ourselves reelected 18:34 < mhroncok> :( 18:34 < decathorpe> mhroncok: ᕕ( ᐛ )ᕗ 18:34 <+sgallagh> Proposal: Replace the CentOS Stream question with "How do you handle disagreements when working as part of a team?" 18:34 < zbyszek> sgallagh: +1 18:34 <+sgallagh> (as suggested by mhayden) 18:34 <+sgallagh> +1 18:35 -!- Guest34324 [~Guest3432@2001:a61:2483:c401:ceb8:afad:3f04:b9e8] has joined #fedora-meeting 18:35 < mhroncok> I don't love "when working as part of a team" in this context but I won't block this. +1 18:35 < nirik> same 18:35 <+sgallagh> mhroncok: Are you suggesting that FESCo doesn't work as a team?! 18:35 <+music[m]> +1, that question may or may not be enlightening in practice but it at least makes some kind of sense 18:36 < mhroncok> Stephen Gallagher: I am suggesting that we often need to handle disagreements from outside FESCo 18:36 < mhroncok> so in that con text, the team is Fedora 18:36 < mhroncok> but it just reads wrong for me, no big deal 18:36 < gotmax> "disagreements within the community"? 18:37 < decathorpe> I trust that bcotton can come up with a sufficiently good alternative formulation ;) 18:38 < mhroncok> sorry for speaking up, let's approve what we have before we bike-shed for another hour 18:38 < zbyszek> Yes, please. 18:38 < decathorpe> it's fine with me. +1 18:38 < nirik> sure, +1 18:38 < dcantrell> +1 18:39 < mhroncok> it's not like people will actually read our answers anyway :D 18:39 < zbyszek> mhroncok: your optimism is shocking 18:39 < mhroncok> it's not me, it's the mush 18:40 <+sgallagh> #agreed Replace the CentOS Stream question with "How do you handle disagreements when working as part of a team?" (+6, 0, -0) 18:40 <+sgallagh> #topic Next week's chair 18:40 < mhroncok> I'll do it 18:40 <+sgallagh> Thanks 18:40 <+sgallagh> #action mhroncok to chair next meeting 18:40 <+sgallagh> #topic Open Floor 18:40 <+sgallagh> Anything for Open Floor? 18:40 <+sgallagh> (Please say no) 18:41 < decathorpe> (no) 18:41 < mhroncok> no 18:41 < zbyszek> If we have a minute 18:41 < zbyszek> OK, just joking. 18:41 < zbyszek> Badly. 18:42 <+sgallagh> #endmeeting 18:42 <+sgallagh> Thanks folks 18:42 -!- pablomh [~pabl...@55.red-83-43-84.dynamicip.rima-tde.net] has quit [Quit: Leaving] 18:42 <+sgallagh> Of course, I think Zodbot died somewhere in there 18:42 < mhroncok> Stephen Gallagher++ 18:42 < mhroncok> I am not afraid 18:42 < decathorpe> sgallagh++ 18:42 < mhroncok> *surprised 18:42 < decathorpe> zodbot-- 18:42 < mhroncok> if I were zodbot, I'd die as well 18:43 < nirik> hum. 18:43 < decathorpe> see you next week o/ 18:44 -!- besser82[m] [besser82@fedora/besser82] has joined #fedora-meeting 18:46 < nirik> zodbot should be back in a min... but it will not remember the meeting. 18:46 < gotmax> I pasted my IRC logs from the FESCo meeting if someone wants: https://paste.sr.ht/~gotmax23/3576b21a357f5da14cc5871925d1f45999197b62 I just saw gotmax's comment after pasting this in ;) Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
