Re: Should a serious bug have made in into bullseye 11.5?
Steve McIntyre wrote: ... > I'm building a new unstable package (2.06-4) right now with Valentin's > patch applied, and once I've uploaded that I'll do a new bullseye > package too. that's great! thank you! :) songbird
Re: Should a serious bug have made in into bullseye 11.5?
Michael Stone wrote: >On Mon, Sep 12, 2022 at 10:10:33AM -0500, David Wright wrote: >>Well, my focus would be on two things: (a) the change in compatibility >>level in debhelper in the middle of stable's lifetime > >That would not have ordinarily happened, and probably shouldn't have >happened in this case. Other non-minimal-security changes were backed >out for bullseye (namely the change to os-prober behavior) but this was >either overlooked or not realized to be a significant change. Usually a >stable update package would be modified from the version in stable >rather than backported from unstable, but in this case there were no >intermediate versions in unstable and it was probably thought safer to >use the package which had been tested in unstable rather than starting >over and potentially introducing a new bug. That probably was even true, >as the problem was identified during the test period on unstable -- but, >unfortunately, the priority of the bug didn't bubble up. I think this is >just one of those cases where mistakes happen (in this case, several >that aligned in an unfortunate way) and regardless of how hard we >(humans) try to avoid them sometimes we don't. Yup, you've nailed it. We've had a stack of security bugs that needed fixing in grub, and I chose to move both buster and bullseye forwards to 2.06 rather than try and backport all the fixes to older releases and hope/pray that they applied sensibly. Grub is very much a moving target and a *huge* codebase with a lot of patches, for historical reasons. I didn't pick up on the packaging bug here, and unfortunately it made it into the bullseye stable release. I tested my grub build on a number of platforms and architectures, but that didn't include Xen. We *really* have a dearth of Xen experience among the maintainers, and that's not helping here. I'm building a new unstable package (2.06-4) right now with Valentin's patch applied, and once I've uploaded that I'll do a new bullseye package too. -- Steve McIntyre, Cambridge, UK.st...@einval.com "We're the technical experts. We were hired so that management could ignore our recommendations and tell us how to do our jobs." -- Mike Andrews
Re: Should a serious bug have made in into bullseye 11.5?
On 9/12/2022 11:36 AM, Tim Woodall wrote: > On Mon, 12 Sep 2022, David Wright wrote: > > > > > AFAICT it had two months in testing without this problem being > > hit and reported. > > > > Unfortunately, g-x-h is probably mostly used on stable or oldstable with > guests running testing. > > I'm not sure if it's possible to run a xen hypervisor in a domu - that > would be an interesting way to run testing. > > Tim. > I think that xen hypervisor running in a domu is called nested virtualization. The Xen project defines support for this as follows [1]: ### x86/Nested PV This means running a Xen hypervisor inside an HVM domain on a Xen system, with support for PV L2 guests only (i.e., hardware virtualization extensions not provided to the guest). Status, x86 Xen HVM: Tech Preview This works, but has performance limitations because the L1 dom0 can only access emulated L1 devices. Xen may also run inside other hypervisors (KVM, Hyper-V, VMWare), but nobody has reported on performance. ### x86/Nested HVM This means providing hardware virtulization support to guest VMs allowing, for instance, a nested Xen to support both PV and HVM guests. It also implies support for other hypervisors, 739 such as KVM, Hyper-V, Bromium, and so on as guests. Status, x86 HVM: Experimental [1] http://xenbits.xen.org/gitweb/?p=xen.git;a=blob_plain;f=SUPPORT.md;hb=HEAD
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, Sep 12, 2022 at 10:10:33AM -0500, David Wright wrote: Well, my focus would be on two things: (a) the change in compatibility level in debhelper in the middle of stable's lifetime That would not have ordinarily happened, and probably shouldn't have happened in this case. Other non-minimal-security changes were backed out for bullseye (namely the change to os-prober behavior) but this was either overlooked or not realized to be a significant change. Usually a stable update package would be modified from the version in stable rather than backported from unstable, but in this case there were no intermediate versions in unstable and it was probably thought safer to use the package which had been tested in unstable rather than starting over and potentially introducing a new bug. That probably was even true, as the problem was identified during the test period on unstable -- but, unfortunately, the priority of the bug didn't bubble up. I think this is just one of those cases where mistakes happen (in this case, several that aligned in an unfortunate way) and regardless of how hard we (humans) try to avoid them sometimes we don't.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, Sep 12, 2022 at 02:32:16PM +, Andy Smith wrote: On Mon, Sep 12, 2022 at 10:15:41AM -0400, Michael Stone wrote: There are automated processes that stop package migration at certain severity levels, but they can't guess that something that was filed at a low level really should have been higher. I think in this case the package was already present in testing and stable-proposed-updates before the bug was found. It was reported as "grave" and bounced between that and "serious". No, it wasn't; as far as I can tell, the bug didn't become "grave" until today. (It looks like someone tried to set it to grave in August but didn't do so properly.) It didn't become "serious" until Sep 6 (after it was already processed into bullseye updates). Also I am not sure if there are the same processes around this for packages going in to stable-proposed-updates. The migrations you speak of are from unstable to testing, and also with "RC" bugs in testing blocking a full release. But stable updates go straight to stable-proposed-updates and I don't know if anyone is watching bugs specific to that when cutting a point release. Yes, release managers would look at a grave bug before issuing a release. Anyway what I am saying is, I'm not sure there is any level of severity setting that would have made a difference in this case, Yes, it likely would have.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, 12 Sep 2022, David Wright wrote: On Mon 12 Sep 2022 at 14:20:59 (+0100), Tim Woodall wrote: The same version also went to oldstable - where it turns out it works fine - so I can see how it could be missed but this was a bug that I feel would have, if necessary, justified delaying the 11.5 release and I wonder what a bug reporter should do in a case like this. AFAICT it had two months in testing without this problem being hit and reported. Unfortunately, g-x-h is probably mostly used on stable or oldstable with guests running testing. I'm not sure if it's possible to run a xen hypervisor in a domu - that would be an interesting way to run testing. Tim.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon 12 Sep 2022 at 14:20:59 (+0100), Tim Woodall wrote: > On Mon, 12 Sep 2022, Andy Smith wrote: > > On Mon, Sep 12, 2022 at 12:00:20PM +, Andy Smith wrote: > > > Obviously, no one desires for there to be bugs, so your question > > > doesn't really make sense. "Should bugs make it into Debian releases"? > > > > Ah, sorry, I think I misunderstood - you are literally asking if the > > presence of a severity "serious" bug in Grub should have prevented > > the whole 11.5 point release happening? > > > > I don't know. The only documentation I can find on the matter is > > about full Debian releases and even that says the bugs would have to > > be apmrked "release-critical" (RC) to block release, so not even > > "critical" may have postponed things. > > > Interesting. I didn't find this bug report until after I'd already > tracked down the culprit package and was ready to file my own bug. Presumably apt-listbugs would have spotted this as the bug was raised to serious last Tuesday, and the point-release was days after that. > > My gut feeling is that there's going to be quite a lot of > > "serious"-level bugs in any point release and that no one works to > > associate these with a recent upload and then prevent that going > > into a new point release. > > > > It still feels more useful to focus on how such problems can be > > avoided in future. I don't think we can explore the release team > > looking at every "serious" bug in every package otherwise they'd > > never get a point release out. Well, my focus would be on two things: (a) the change in compatibility level in debhelper in the middle of stable's lifetime, and (b) on why grub-xen-host has fallen behind on the debhelper compatibility level that it supports. I don't know enought to comment on (a). It would seem to be a simple matter for g-x-h to have used the exception mechanism in debhelper rather than to rely on its guesswork. But I also guess that it would be easy to miss the change when it occurred (2017) because xen in this configuration is relatively little used and therefore less resourced. But if g-x-h had raised the compatibility level in a more timely manner, then I think this bug would have had to escape notice for ~two years in bullseye-as-testing, rather than the two months in bookworm/testing in order to survive in stable. This bug illustrates the danger of sticking with an ancient compatibility level. > Agreed. While I tend to try to file bugs at the lowest severity that can > be justified, I know that others go the other way. This is one I'd > probably have filed as Grave or even Critical. (I see it's now been > bumped to Grave) > > It just felt wrong to me that this bug (and version bump of the > package) could go to stable without someone at least acknowleging the > bug. AFAICT there's no fundamental reason it needed to go out. Perhaps because this version of Grub fixes seven CVEs? > If it was > that the reporter should have marked it grave or critical then fair > enough, just unfortunate that they didn't. The reporter is often not best-placed to make that judgment. > The same version also went to oldstable - where it turns out it works > fine - so I can see how it could be missed but this was a bug that I > feel would have, if necessary, justified delaying the 11.5 release and I > wonder what a bug reporter should do in a case like this. AFAICT it had two months in testing without this problem being hit and reported. > Fortunately, from my PoV, it came with a kernel update and at the > weekend, so I rebooted and had time to investigate what was going on. > Otherwise I might have been blissfully unaware until a power-cut... > > Unfortunately, on Saturday morning I'd removed pvshim=1 from the last of > my guests and restarted them (successfully) so I wasn't 100% sure it > wasn't something I'd done wrong. I notice that others are now suggesting apt-listbugs. I've seen real show-stoppers in its reports on occasions, but they've usually not applied to my systems (different architectures, or combination of packages etc), determined by inspecting the already downloaded package. A real boon though. Cheers, David.
Re: Should a serious bug have made in into bullseye 11.5?
Hello, On Mon, Sep 12, 2022 at 10:15:41AM -0400, Michael Stone wrote: > There are automated processes that stop package migration at > certain severity levels, but they can't guess that something that > was filed at a low level really should have been higher. I think in this case the package was already present in testing and stable-proposed-updates before the bug was found. It was reported as "grave" and bounced between that and "serious". Also I am not sure if there are the same processes around this for packages going in to stable-proposed-updates. The migrations you speak of are from unstable to testing, and also with "RC" bugs in testing blocking a full release. But stable updates go straight to stable-proposed-updates and I don't know if anyone is watching bugs specific to that when cutting a point release. Anyway what I am saying is, I'm not sure there is any level of severity setting that would have made a difference in this case, only perhaps a lot of people experiencing the problem in time and shouting about it (or automated testing to catch it). Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, Sep 12, 2022 at 02:20:59PM +0100, Tim Woodall wrote: Agreed. While I tend to try to file bugs at the lowest severity that can be justified, I know that others go the other way. This is one I'd probably have filed as Grave or even Critical. (I see it's now been bumped to Grave) If it's something that should be a show-stopper it should be filed at a high severity and downgraded if necessary. There are automated processes that stop package migration at certain severity levels, but they can't guess that something that was filed at a low level really should have been higher.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, 12 Sep 2022, David wrote: On Mon, 12 Sept 2022 at 23:21, Tim Woodall wrote: It just felt wrong to me that this bug (and version bump of the package) could go to stable without someone at least acknowleging the bug. AFAICT there's no fundamental reason it needed to go out. If it was that the reporter should have marked it grave or critical then fair enough, just unfortunate that they didn't. Bugs are occasionally present on stable packages. I have apt-listbugs installed. By default, it warns about critical,grave,serious bugs on the package, and asks for confirmation before installing the package. This provides an opportunity to read the bug and make a decision about whether to install the buggy package, or hold the package at a previous version. Thanks, I will investigate that. Tim.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, 12 Sept 2022 at 23:21, Tim Woodall wrote: > On Mon, 12 Sep 2022, Andy Smith wrote: > > On Mon, Sep 12, 2022 at 12:00:20PM +, Andy Smith wrote: > >> Obviously, no one desires for there to be bugs, so your question > >> doesn't really make sense. "Should bugs make it into Debian releases"? > > > > Ah, sorry, I think I misunderstood - you are literally asking if the > > presence of a severity "serious" bug in Grub should have prevented > > the whole 11.5 point release happening? > > > > I don't know. The only documentation I can find on the matter is > > about full Debian releases and even that says the bugs would have to > > be apmrked "release-critical" (RC) to block release, so not even > > "critical" may have postponed things. > > > Interesting. I didn't find this bug report until after I'd already > tracked down the culprit package and was ready to file my own bug. > > > My gut feeling is that there's going to be quite a lot of > > "serious"-level bugs in any point release and that no one works to > > associate these with a recent upload and then prevent that going > > into a new point release. > > > > It still feels more useful to focus on how such problems can be > > avoided in future. I don't think we can explore the release team > > looking at every "serious" bug in every package otherwise they'd > > never get a point release out. > > > Agreed. While I tend to try to file bugs at the lowest severity that can > be justified, I know that others go the other way. This is one I'd > probably have filed as Grave or even Critical. (I see it's now been > bumped to Grave) > > It just felt wrong to me that this bug (and version bump of the > package) could go to stable without someone at least acknowleging the > bug. AFAICT there's no fundamental reason it needed to go out. If it was > that the reporter should have marked it grave or critical then fair > enough, just unfortunate that they didn't. Bugs are occasionally present on stable packages. I have apt-listbugs installed. By default, it warns about critical,grave,serious bugs on the package, and asks for confirmation before installing the package. This provides an opportunity to read the bug and make a decision about whether to install the buggy package, or hold the package at a previous version.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, 12 Sep 2022, Andy Smith wrote: On Mon, Sep 12, 2022 at 12:00:20PM +, Andy Smith wrote: Obviously, no one desires for there to be bugs, so your question doesn't really make sense. "Should bugs make it into Debian releases"? Ah, sorry, I think I misunderstood - you are literally asking if the presence of a severity "serious" bug in Grub should have prevented the whole 11.5 point release happening? I don't know. The only documentation I can find on the matter is about full Debian releases and even that says the bugs would have to be apmrked "release-critical" (RC) to block release, so not even "critical" may have postponed things. Interesting. I didn't find this bug report until after I'd already tracked down the culprit package and was ready to file my own bug. My gut feeling is that there's going to be quite a lot of "serious"-level bugs in any point release and that no one works to associate these with a recent upload and then prevent that going into a new point release. It still feels more useful to focus on how such problems can be avoided in future. I don't think we can explore the release team looking at every "serious" bug in every package otherwise they'd never get a point release out. Agreed. While I tend to try to file bugs at the lowest severity that can be justified, I know that others go the other way. This is one I'd probably have filed as Grave or even Critical. (I see it's now been bumped to Grave) It just felt wrong to me that this bug (and version bump of the package) could go to stable without someone at least acknowleging the bug. AFAICT there's no fundamental reason it needed to go out. If it was that the reporter should have marked it grave or critical then fair enough, just unfortunate that they didn't. The same version also went to oldstable - where it turns out it works fine - so I can see how it could be missed but this was a bug that I feel would have, if necessary, justified delaying the 11.5 release and I wonder what a bug reporter should do in a case like this. Fortunately, from my PoV, it came with a kernel update and at the weekend, so I rebooted and had time to investigate what was going on. Otherwise I might have been blissfully unaware until a power-cut... Unfortunately, on Saturday morning I'd removed pvshim=1 from the last of my guests and restarted them (successfully) so I wasn't 100% sure it wasn't something I'd done wrong. Tim.
Re: Should a serious bug have made in into bullseye 11.5?
On Mon, Sep 12, 2022 at 12:00:20PM +, Andy Smith wrote: > Obviously, no one desires for there to be bugs, so your question > doesn't really make sense. "Should bugs make it into Debian releases"? Ah, sorry, I think I misunderstood - you are literally asking if the presence of a severity "serious" bug in Grub should have prevented the whole 11.5 point release happening? I don't know. The only documentation I can find on the matter is about full Debian releases and even that says the bugs would have to be apmrked "release-critical" (RC) to block release, so not even "critical" may have postponed things. My gut feeling is that there's going to be quite a lot of "serious"-level bugs in any point release and that no one works to associate these with a recent upload and then prevent that going into a new point release. It still feels more useful to focus on how such problems can be avoided in future. I don't think we can explore the release team looking at every "serious" bug in every package otherwise they'd never get a point release out. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Should a serious bug have made in into bullseye 11.5?
On Sun, Sep 11, 2022 at 09:23:17PM +0100, Tim Woodall wrote: > Should https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017944 have > made it into debian 11.5? I thought serious bugs shouldn't make it into > stable? I'm trying to read your email charitably in the sense that you are wondering how best this sort of thing can be prevented in future, rather than as an angry response that seeks to assign blame to volunteers. Obviously, no one desires for there to be bugs, so your question doesn't really make sense. "Should bugs make it into Debian releases"? The various severities assigned to bugs are used to help direct available resources and at a high enough level they can block a release until they are addressed. This can only work AFTER the bug has been discovered. So the assigned severity of the bug has no relation to a release that already happened. If you mean, "how can a release have happened that included a bug as severe as this?" the answer is simply (lack of) available resources for testing. Ideally, every time there was a commit to the grub package there would be automated CI jobs testing a boot of PV and PVH mode Xen. But there's not, because nobody has got around to doing that. Is that something you could help with? Xen is a really really niche use case so I'm not surprised that a change in Grub was not tested for the Xen use case, neither by the Grub maintainers nor by the Xen maintainers. I don't think there is likely to be any other distribution that does better for that use case, either. Maybe the ones you pay to integrate this stuff together, like Citrix XenServer. Upstream Xen does have a battery of CI tests I believe, but I don't think that the Debian Xen team does, so again the packages produced by the Debian Xen team are only being tested to the extent that the maintainers and volunteers are doing their own testing. Clearly bugs WILL make it into Debian releases. It is worth thinking about ways to reduce the chance of this happening. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting signature.asc Description: PGP signature
Re: Should a serious bug have made in into bullseye 11.5?
On Mon 12 Sep 2022 at 03:36:46 (+0100), Tim Woodall wrote: > On Sun, 11 Sep 2022, Greg Wooledge wrote: > > On Sun, Sep 11, 2022 at 09:23:17PM +0100, Tim Woodall wrote: > > > Should https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017944 have > > > made it into debian 11.5? I thought serious bugs shouldn't make it into > > > stable? > > > > Bugs have to be discovered and reported. If nobody found this during > > the bullseye testing cycle, then it's quite likely bullseye would ship > > with this bug. > > > > I note that the bug report was reported August 22, 2022, over one year > > after the bullseye release. Maybe this package just doesn't have many > > users, or the bug didn't affect most of its users. > > > > > The buggy package was added to bullseye with the 11.5 point release on > 2022-09-10. It wasn't present in 11.4 whose version of this package > worked. It was added to proposed-updates on 2022-08-06. > > Based on the comments in thw bug it looks like the breakage was due to > the changing to use debhelper 13 instead of 10. One way to read this is that a misfeature had been in grub-xen-host since very early on. If it relies on not stripping the symbols from any files, then those files' names should really have been placed in the exception list that gets handed to dh_strip: -Xitem, --exclude=item Exclude files that contain item anywhere in their filename from being stripped. You may use this option multiple times to build up a list of things to exclude. As it was, there's was reliance on debhelper's guesswork, something that had been reported as poor in 1999 (#35733), and improved in 2017 (protected by compatibility level 10). It's beyond me to work out exactly when compatibility 10 becomes deprecated, how strong deprecation actually is, what pressure there is to increase the level in grub-xen-host, and why its level had fallen so far behind. Cheers, David.
Re: Should a serious bug have made in into bullseye 11.5?
On Sun, 11 Sep 2022, Greg Wooledge wrote: On Sun, Sep 11, 2022 at 09:23:17PM +0100, Tim Woodall wrote: Should https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017944 have made it into debian 11.5? I thought serious bugs shouldn't make it into stable? Bugs have to be discovered and reported. If nobody found this during the bullseye testing cycle, then it's quite likely bullseye would ship with this bug. I note that the bug report was reported August 22, 2022, over one year after the bullseye release. Maybe this package just doesn't have many users, or the bug didn't affect most of its users. The buggy package was added to bullseye with the 11.5 point release on 2022-09-10. It wasn't present in 11.4 whose version of this package worked. It was added to proposed-updates on 2022-08-06. Based on the comments in thw bug it looks like the breakage was due to the changing to use debhelper 13 instead of 10.
Re: Should a serious bug have made in into bullseye 11.5?
On Sun, Sep 11, 2022 at 09:23:17PM +0100, Tim Woodall wrote: > Should https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017944 have > made it into debian 11.5? I thought serious bugs shouldn't make it into > stable? Bugs have to be discovered and reported. If nobody found this during the bullseye testing cycle, then it's quite likely bullseye would ship with this bug. I note that the bug report was reported August 22, 2022, over one year after the bullseye release. Maybe this package just doesn't have many users, or the bug didn't affect most of its users.
