Bug#880554: [Pkg-xen-devel] Bug#880554: #880554: max grant frames problem
On 7/18/19 1:30 AM, Hans van Kranenburg wrote: > Hi, > > On 10/23/18 7:34 PM, Ian Jackson wrote: >> Control: retitle -1 max grant frames problem (domu freeze with >> linux-image-4.9.0-4-amd64) >> Control: severity -1 important >> Control: reassign -1 src:xen 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 > > my last comment in this bts bug was about: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=29d11cfd8698038b87458ba4d1329b9da81150a5 > > ..which is in since linux 4.13-rc2, and buster has 4.19+ > > Is there anyone who would wants to try reproduce the max grant frames > problem on buster with Xen 4.11 and Linux 4.19 dom0/domU? > > The 'xen/grant-table: max_grant_frames reached' should show up on the > serial console. I'd like to see a test report of it actually happening. I actually just did this, by putting max_grant_frames = 4 in a domU config file and starting it (Linux 4.19 domU on Xen 4.11): Welcome to Debian GNU/Linux 10 (buster)! [5.499058] systemd[1]: Set hostname to . [5.552968] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.554012] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.555858] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.556950] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.557082] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.557295] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.557636] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.558960] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [5.559800] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=3 req_entries=1 [6.014291] gnttab_expand: 159 callbacks suppressed [6.014296] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.014351] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=8 [6.033683] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.055013] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.055729] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=26 [6.060256] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.077000] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.109760] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.138126] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 [6.148626] xen:grant_table: xen/grant-table: max_grant_frames reached cur=4 extra=1 limit=4 gnttab_free_count=0 req_entries=3 Yay. Better info for the users! Also, there's a patch in review that can improve the situation: https://lists.xenproject.org/archives/html/xen-devel/2019-11/msg01607.html The biggest annoyance in our Xen 4.11 now is that the default value for the hypervisor command line of gnttab_max_frames is raised to 64 from 32 a while ago, but the toolstack overwrites this again with a default of 32. The patch attempts to fix that. Hans
Bug#880554: [Pkg-xen-devel] Bug#880554: #880554: max grant frames problem
Hi, On 10/23/18 7:34 PM, Ian Jackson wrote: > Control: retitle -1 max grant frames problem (domu freeze with > linux-image-4.9.0-4-amd64) > Control: severity -1 important > Control: reassign -1 src:xen 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 my last comment in this bts bug was about: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=29d11cfd8698038b87458ba4d1329b9da81150a5 ..which is in since linux 4.13-rc2, and buster has 4.19+ Is there anyone who would wants to try reproduce the max grant frames problem on buster with Xen 4.11 and Linux 4.19 dom0/domU? The 'xen/grant-table: max_grant_frames reached' should show up on the serial console. I'd like to see a test report of it actually happening. No further adjustments/fixes will go into the Stretch Xen packages at this stage. Having better documentation about how to set hypervisor and guest options to deal with all of this is still a TODO. I would really like to get some people together to start cleaning out the whole Xen related wiki section for Debian, and actually provide some helpful content, including FAQ stuff like max grants, PVH, PVH+grub etc... Whoever would want to participate in that, just reply a Yay! Doing documentation work might seem boring, but it's write once, read many all the way. Hans
Bug#880554: [Pkg-xen-devel] Bug#880554: #880554: max grant frames problem
Hi, Our Buster TODO [1] has a TODO item for me about adding a paragraph in the "Known Issues" section of the Debian README / NEWS / or wherever it will go about the grant frames issue. It would be very nice to have some WARN_ONCE code in the linux kernel exactly at the place where this issue happens, but nobody has been adding that yet, so for now, if this happens, domUs will just still hang without meaningful stuff in dmesg. I also have a suspicion that this issue will be showing up less when using Xen 4.11+ and Linux 4.19+ in dom0 and domU (so, yay, upgrade all the things!). Newer Linux kernel versions have patches that also actually release frame stuff when it's no longer actually in use, which might end the 'only-going-up' behaviour of the number we see. Anyway, I'm planning to add some "known issue" documentation about this to the Xen 4.11 packaging, together with a clear short description of what to change where in what configuration (because in Xen 4.11+ it's different than in 4.4 or 4.8 again), and then mark this bts as closed while that upload happens. Digging around in kernel code to find out where this happens and adding this WARN_ONCE is still a nice kernel coding exercise, but I don't want this bts to track if that ends up on top of my TODO list or not. Hans [1] https://salsa.debian.org/xen-team/debian-xen/issues/24
Bug#880554: #880554: max grant frames problem
Control: retitle -1 max grant frames problem (domu freeze with linux-image-4.9.0-4-amd64) Control: severity -1 important Control: reassign -1 src:xen 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 Just gardening here. (i) Bug title should mention grant frames. (ii) This does not affect all use cases and is not, IMO, RC. Although we should certainly see if we can improve it. (iii) Britney is confused because the bug was reassigned to xen-hypervisor-4.8-amd64 and thinks that updating to the 4.11-based packages from sid would help reduce RC bugs since they lack that .deb. This is wrong, of course. For this reason in general bugs should be reported against src:xen rather than against binary packages with Xen versions in their package name. Ian. -- Ian JacksonThese opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.
