Re: Increasing domain memory beyond initial maxmem
On Wed, Apr 06, 2022 at 07:13:18AM +0200, Juergen Gross wrote:
> On 05.04.22 18:24, Marek Marczykowski-Górecki wrote:
> > On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
> > > Hi Marek,
> > >
> > > On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
> > > > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> > > > > Maybe some kernel config differences, or other udev rules (memory
> > > > > onlining
> > > > > is done via udev in my guest)?
> > > > >
> > > > > I'm seeing:
> > > > >
> > > > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > > > CONFIG_MEMORY_HOTPLUG=y
> > > > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> > > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> > > >
> > > > I have:
> > > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > > CONFIG_MEMORY_HOTPLUG=y
> > > > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> > > >
> > > > Not sure if relevant, but I also have:
> > > > CONFIG_XEN_UNPOPULATED_ALLOC=y
> > > >
> > > > on top of that, I have a similar udev rule too:
> > > >
> > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline",
> > > > ATTR{state}="online"
> > > >
> > > > But I don't think they are conflicting.
> > > >
> > > > > What type of guest are you using? Mine was a PVH guest.
> > > >
> > > > PVH here too.
> > >
> > > Would you like to try the attached patch? It seemed to work for me.
> >
> > Unfortunately it doesn't help, now the behavior is different:
> >
> > Initially guest started with 800M:
> >
> > [root@personal ~]# free -m
> >totalusedfree shared buff/cache
> > available
> > Mem:740 223 272 2 243
> > 401
> > Swap: 1023 01023
> >
> > Then increased:
> >
> > [root@dom0 ~]$ xl mem-max personal 2048
> > [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid
> > personal)/memory/static-max $((2048*1024))
> > [root@dom0 ~]$ xl mem-set personal 2000
> >
> > And guest shows now only a little more memory, but not full 2000M:
> >
> > [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
> > [ 37.658206] Fallback order for Node 0: 0
> > [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages:
> > 175889
> > [ 37.658233] Policy zone: Normal
> >
> > [root@personal ~]#
> > [root@personal ~]# free -m
> >totalusedfree shared buff/cache
> > available
> > Mem:826 245 337 2 244
> > 462
> > Swap: 1023 01023
> >
> >
> > I've applied the patch on top of 5.16.18. If you think 5.17 would make a
> > difference, I can try that too.
>
> Hmm, weird.
>
> Can you please post the output of
>
> cat /proc/buddyinfo
> cat /proc/iomem
>
> in the guest before and after the operations?
Ok, that was a stupid mistake on my side - I've run out of host memory.
With that fixed, it seems to work, on 5.16.18 too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
signature.asc
Description: PGP signature
Re: Increasing domain memory beyond initial maxmem
On 05.04.22 18:24, Marek Marczykowski-Górecki wrote:
On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
Hi Marek,
On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
Maybe some kernel config differences, or other udev rules (memory onlining
is done via udev in my guest)?
I'm seeing:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
I have:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
Not sure if relevant, but I also have:
CONFIG_XEN_UNPOPULATED_ALLOC=y
on top of that, I have a similar udev rule too:
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
But I don't think they are conflicting.
What type of guest are you using? Mine was a PVH guest.
PVH here too.
Would you like to try the attached patch? It seemed to work for me.
Unfortunately it doesn't help, now the behavior is different:
Initially guest started with 800M:
[root@personal ~]# free -m
totalusedfree shared buff/cache
available
Mem:740 223 272 2 243
401
Swap: 1023 01023
Then increased:
[root@dom0 ~]$ xl mem-max personal 2048
[root@dom0 ~]$ xenstore-write /local/domain/$(xl domid
personal)/memory/static-max $((2048*1024))
[root@dom0 ~]$ xl mem-set personal 2000
And guest shows now only a little more memory, but not full 2000M:
[root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
[ 37.658206] Fallback order for Node 0: 0
[ 37.658219] Built 1 zonelists, mobility grouping on. Total pages:
175889
[ 37.658233] Policy zone: Normal
[root@personal ~]#
[root@personal ~]# free -m
totalusedfree shared buff/cache
available
Mem:826 245 337 2 244
462
Swap: 1023 01023
I've applied the patch on top of 5.16.18. If you think 5.17 would make a
difference, I can try that too.
Hmm, weird.
Can you please post the output of
cat /proc/buddyinfo
cat /proc/iomem
in the guest before and after the operations?
Juergen
OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
Re: Increasing domain memory beyond initial maxmem
On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
> Hi Marek,
>
> On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
> > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> > > Maybe some kernel config differences, or other udev rules (memory onlining
> > > is done via udev in my guest)?
> > >
> > > I'm seeing:
> > >
> > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > CONFIG_MEMORY_HOTPLUG=y
> > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> >
> > I have:
> > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > CONFIG_MEMORY_HOTPLUG=y
> > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> >
> > Not sure if relevant, but I also have:
> > CONFIG_XEN_UNPOPULATED_ALLOC=y
> >
> > on top of that, I have a similar udev rule too:
> >
> > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline",
> > ATTR{state}="online"
> >
> > But I don't think they are conflicting.
> >
> > > What type of guest are you using? Mine was a PVH guest.
> >
> > PVH here too.
>
> Would you like to try the attached patch? It seemed to work for me.
Unfortunately it doesn't help, now the behavior is different:
Initially guest started with 800M:
[root@personal ~]# free -m
totalusedfree shared buff/cache
available
Mem:740 223 272 2 243
401
Swap: 1023 01023
Then increased:
[root@dom0 ~]$ xl mem-max personal 2048
[root@dom0 ~]$ xenstore-write /local/domain/$(xl domid
personal)/memory/static-max $((2048*1024))
[root@dom0 ~]$ xl mem-set personal 2000
And guest shows now only a little more memory, but not full 2000M:
[root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
[ 37.658206] Fallback order for Node 0: 0
[ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889
[ 37.658233] Policy zone: Normal
[root@personal ~]#
[root@personal ~]# free -m
totalusedfree shared buff/cache
available
Mem:826 245 337 2 244
462
Swap: 1023 01023
I've applied the patch on top of 5.16.18. If you think 5.17 would make a
difference, I can try that too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
signature.asc
Description: PGP signature
Re: Increasing domain memory beyond initial maxmem
Hi Marek,
On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
Maybe some kernel config differences, or other udev rules (memory onlining
is done via udev in my guest)?
I'm seeing:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
I have:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
Not sure if relevant, but I also have:
CONFIG_XEN_UNPOPULATED_ALLOC=y
on top of that, I have a similar udev rule too:
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
But I don't think they are conflicting.
What type of guest are you using? Mine was a PVH guest.
PVH here too.
Would you like to try the attached patch? It seemed to work for me.
Juergen
From a605232115a9c3d3f8103d0833b149ff22956c4b Mon Sep 17 00:00:00 2001
From: Juergen Gross
To: linux-ker...@vger.kernel.org
Cc: Boris Ostrovsky
Cc: Juergen Gross
Cc: Stefano Stabellini
Cc: xen-devel@lists.xenproject.org
Date: Tue, 5 Apr 2022 12:43:41 +0200
Subject: [PATCH] xen/balloon: fix page onlining when populating new zone
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When onlining a new memory page in a guest the Xen balloon driver is
adding it to the ballooned pages instead making it available to be
used immediately. This is meant to enable to add a new upper memory
limit to a guest via hotplugging memory, without having to assign the
new memory in one go.
In case the upper memory limit will be raised above 4G, the new memory
will populate the ZONE_NORMAL memory zone, which wasn't populated
before. The newly populated zone won't be added to the list of zones
looked at by the page allocator though, as only zones with available
memory are being added, and the memory isn't yet available as it is
ballooned out.
This will result in the new memory being assigned to the guest, but
without the allocator being able to use it.
When running as a PV guest the situation is even worse: when having
been started with less memory than allowed, and the upper limit being
lower than 4G, ballooning up will have the same effect as hotplugging
new memory. This is due to the usage of the zone device functionality
since commit 9e2369c06c8a ("xen: add helpers to allocate unpopulated
memory") for creating mappings of other guest's pages, which as a side
effect is being used for PV guest ballooning, too.
Fix this by checking in xen_online_page() whether the new memory page
will be the first in a new zone. If this is the case, add another page
to the balloon and use the first memory page of the new chunk as a
replacement for this now ballooned out page. This will result in the
newly populated zone containing one page being available for the page
allocator, which in turn will lead to the zone being added to the
allocator.
Cc: sta...@vger.kernel.org
Fixes: 9e2369c06c8a ("xen: add helpers to allocate unpopulated memory")
Reported-by: Marek Marczykowski-Górecki
Signed-off-by: Juergen Gross
---
drivers/xen/balloon.c | 72 ++-
1 file changed, 65 insertions(+), 7 deletions(-)
diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
index dfe26fa17e95..f895c54c4c65 100644
--- a/drivers/xen/balloon.c
+++ b/drivers/xen/balloon.c
@@ -355,14 +355,77 @@ static enum bp_state reserve_additional_memory(void)
return BP_ECANCELED;
}
+static struct page *alloc_page_for_balloon(gfp_t gfp)
+{
+ struct page *page;
+
+ page = alloc_page(gfp);
+ if (page == NULL)
+ return NULL;
+
+ adjust_managed_page_count(page, -1);
+ xenmem_reservation_scrub_page(page);
+
+ return page;
+}
+
+static void add_page_to_balloon(struct page *page)
+{
+ xenmem_reservation_va_mapping_reset(1, );
+ balloon_append(page);
+}
+
static void xen_online_page(struct page *page, unsigned int order)
{
unsigned long i, size = (1 << order);
unsigned long start_pfn = page_to_pfn(page);
struct page *p;
+ struct zone *zone;
pr_debug("Online %lu pages starting at pfn 0x%lx\n", size, start_pfn);
mutex_lock(_mutex);
+ zone = page_zone(pfn_to_page(start_pfn));
+
+ /*
+ * In case a new memory zone is going to be populated, we need to
+ * ensure at least one page is made available for the memory allocator.
+ * As the number of pages per zone is updated only after a batch of
+ * pages having been added, use the number of managed pages as an
+ * additional indicator for a new zone.
+ * Otherwise this zone won't be added to the zonelist resulting in the
+ * zone's memory not usable by the kernel.
+ * Add an already valid page to the balloon and replace it with the
+ * first page of the to be added new
Re: Increasing domain memory beyond initial maxmem
On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> Maybe some kernel config differences, or other udev rules (memory onlining
> is done via udev in my guest)?
>
> I'm seeing:
>
> # zgrep MEMORY_HOTPLUG /proc/config.gz
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG=y
> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
I have:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
Not sure if relevant, but I also have:
CONFIG_XEN_UNPOPULATED_ALLOC=y
on top of that, I have a similar udev rule too:
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
But I don't think they are conflicting.
> What type of guest are you using? Mine was a PVH guest.
PVH here too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
signature.asc
Description: PGP signature
Re: Increasing domain memory beyond initial maxmem
On 31.03.22 14:01, Marek Marczykowski-Górecki wrote: On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote: On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: Hi, I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase domain memory beyond initial maxmem, but I hit few issues. A little context: domains in Qubes OS start with rather little memory (400MB by default) but maxmem set higher (4GB by default). Then, there is qmemman daemon, that adjust balloon targets for domains, based on (among other things) demand reported by the domains themselves. There is also a little swap, to mitigate qmemman latency (few hundreds ms at worst). This initial memory < maxmmem in case of PVH / HVM makes use of PoD which I'm trying to get rid of. But also, IIUC Linux will waste some memory for bookkeeping based on maxmem, not actually usable memory. First issue: after using `xl mem-max`, `xl mem-set` still refuses to increase memory more than initial maxmem. That's because xl mem-max does not update 'memory/static-max' xenstore node. This one is easy to work around. Then, the actual hotplug fails on the domU side with: [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 [ 50.004792] Call Trace: [ 50.004799] [ 50.004808] dump_stack_lvl+0x48/0x5e [ 50.004821] warn_alloc+0x162/0x190 [ 50.004832] ? __alloc_pages+0x1fa/0x230 [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 [ 50.004868] vmemmap_populate+0x9e/0x16c [ 50.004878] __populate_section_memmap+0x6a/0xb1 [ 50.004890] section_activate+0x20a/0x278 [ 50.004901] sparse_add_section+0x70/0x160 [ 50.004911] __add_pages+0xc3/0x150 [ 50.004921] add_pages+0x12/0x60 [ 50.004931] add_memory_resource+0x12b/0x320 [ 50.004943] reserve_additional_memory+0x10c/0x150 [ 50.004958] balloon_thread+0x206/0x360 [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 [ 50.004991] kthread+0x16b/0x190 [ 50.005001] ? set_kthread_struct+0x40/0x40 [ 50.005011] ret_from_fork+0x22/0x30 [ 50.005022] Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 After the above, `free` reports correct size (1GB in this case), but that memory seems to be unusable really. "used" is kept low, and soon OOM-killer kicks in. I know the initial 400MB is not much for a full Linux, with X11 etc. But I wouldn't expect it to fail this way when _adding_ memory. I've tried also with initial 800MB. In this case, I do not get "alloc failure" any more, but monitoring `free`, the extra memory still doesn't seem to be used. Any ideas? I can't reproduce that. I started a guest with 8GB of memory, in the guest I'm seeing: # uname -a Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 CET 2022 x86_64 x86_64 x86_64 GNU/Linux # free total used free shared buff/cache available Mem: 817826071628 8023300 8560 83332 8010196 Swap: 20971320 2097132 Then I'm raising the memory for the guest in dom0: # xl list NameID Mem VCPUs State Time(s) Domain-0 0 2634 8 r-1016.5 Xenstore 131 1 -b 0.9 sle15sp1 3 8190 6 -b 184.6 # xl mem-max 3 1 # xenstore-write /local/domain/3/memory/static-max 1024 # xl mem-set 3 1 # xl list NameID Mem VCPUs State Time(s) Domain-0 0 2634 8 r-1018.5 Xenstore 131 1 -b 1.0 sle15sp1 3 1 6 -b 186.7 In the guest I get now: # free total used free shared buff/cache available Mem: 10031700 110904 9734172 8560 186624 9814344 Swap: 20971320 2097132 And after using lots of memory via a ramdisk: # free total used free shared buff/cache available Mem: 10031700 116660 1663840 7181776 8251200 2635372 Swap: 20971320 2097132 You can see buff/cache is now larger than the initial total memory and free is lower than the added memory amount. Hmm, I have a different behavior: I'm starting with 800M # uname -a Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # free -m totalusedfree shared buff/cache available Mem:740 209 278 2 252 415 Swap: 1023 01023 Then raising to ~2GB: [root@dom0 ~]# xl list NameID Mem VCPUs State Time(s) Domain-0 0
Re: Increasing domain memory beyond initial maxmem
On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote: > On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: > > Hi, > > > > I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase > > domain memory beyond initial maxmem, but I hit few issues. > > > > A little context: domains in Qubes OS start with rather little memory > > (400MB by default) but maxmem set higher (4GB by default). Then, there is > > qmemman daemon, that adjust balloon targets for domains, based on (among > > other things) demand reported by the domains themselves. There is also a > > little swap, to mitigate qmemman latency (few hundreds ms at worst). > > This initial memory < maxmmem in case of PVH / HVM makes use of PoD > > which I'm trying to get rid of. But also, IIUC Linux will waste some > > memory for bookkeeping based on maxmem, not actually usable memory. > > > > First issue: after using `xl mem-max`, `xl mem-set` still refuses to > > increase memory more than initial maxmem. That's because xl mem-max does > > not update 'memory/static-max' xenstore node. This one is easy to work > > around. > > > > Then, the actual hotplug fails on the domU side with: > > > > [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, > > mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), > > nodemask=(null),cpuset=/,mems_allowed=0 > > [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted > > 5.16.15-1.37.fc32.qubes.x86_64 #1 > > [ 50.004792] Call Trace: > > [ 50.004799] > > [ 50.004808] dump_stack_lvl+0x48/0x5e > > [ 50.004821] warn_alloc+0x162/0x190 > > [ 50.004832] ? __alloc_pages+0x1fa/0x230 > > [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 > > [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 > > [ 50.004868] vmemmap_populate+0x9e/0x16c > > [ 50.004878] __populate_section_memmap+0x6a/0xb1 > > [ 50.004890] section_activate+0x20a/0x278 > > [ 50.004901] sparse_add_section+0x70/0x160 > > [ 50.004911] __add_pages+0xc3/0x150 > > [ 50.004921] add_pages+0x12/0x60 > > [ 50.004931] add_memory_resource+0x12b/0x320 > > [ 50.004943] reserve_additional_memory+0x10c/0x150 > > [ 50.004958] balloon_thread+0x206/0x360 > > [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 > > [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 > > [ 50.004991] kthread+0x16b/0x190 > > [ 50.005001] ? set_kthread_struct+0x40/0x40 > > [ 50.005011] ret_from_fork+0x22/0x30 > > [ 50.005022] > > > > Full dmesg: > > https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 > > > > After the above, `free` reports correct size (1GB in this case), but > > that memory seems to be unusable really. "used" is kept low, and soon > > OOM-killer kicks in. > > > > I know the initial 400MB is not much for a full Linux, with X11 etc. But > > I wouldn't expect it to fail this way when _adding_ memory. > > > > I've tried also with initial 800MB. In this case, I do not get "alloc > > failure" any more, but monitoring `free`, the extra memory still doesn't > > seem to be used. > > > > Any ideas? > > > > I can't reproduce that. > > I started a guest with 8GB of memory, in the guest I'm seeing: > > # uname -a > Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 > CET 2022 x86_64 x86_64 x86_64 GNU/Linux > # free > total used free shared buff/cache available > Mem: 817826071628 8023300 8560 83332 8010196 > Swap: 20971320 2097132 > > Then I'm raising the memory for the guest in dom0: > > # xl list > NameID Mem VCPUs State Time(s) > Domain-0 0 2634 8 r-1016.5 > Xenstore 131 1 -b 0.9 > sle15sp1 3 8190 6 -b 184.6 > # xl mem-max 3 1 > # xenstore-write /local/domain/3/memory/static-max 1024 > # xl mem-set 3 1 > # xl list > NameID Mem VCPUs State Time(s) > Domain-0 0 2634 8 r-1018.5 > Xenstore 131 1 -b 1.0 > sle15sp1 3 1 6 -b 186.7 > > In the guest I get now: > > # free > total used free shared buff/cache available > Mem: 10031700 110904 9734172 8560 186624 9814344 > Swap: 20971320 2097132 > > And after using lots of memory via a ramdisk: > > # free > total used free shared buff/cache available > Mem: 10031700 116660 1663840 7181776 8251200 2635372 > Swap: 20971320 2097132 > > You can see buff/cache is now larger than the initial total memory > and free is lower than the added memory amount. Hmm, I have a different behavior: I'm starting with 800M # uname -a Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # free -m totalusedfree shared buff/cache available Mem:
Re: Increasing domain memory beyond initial maxmem
On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: Hi, I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase domain memory beyond initial maxmem, but I hit few issues. A little context: domains in Qubes OS start with rather little memory (400MB by default) but maxmem set higher (4GB by default). Then, there is qmemman daemon, that adjust balloon targets for domains, based on (among other things) demand reported by the domains themselves. There is also a little swap, to mitigate qmemman latency (few hundreds ms at worst). This initial memory < maxmmem in case of PVH / HVM makes use of PoD which I'm trying to get rid of. But also, IIUC Linux will waste some memory for bookkeeping based on maxmem, not actually usable memory. First issue: after using `xl mem-max`, `xl mem-set` still refuses to increase memory more than initial maxmem. That's because xl mem-max does not update 'memory/static-max' xenstore node. This one is easy to work around. Then, the actual hotplug fails on the domU side with: [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 [ 50.004792] Call Trace: [ 50.004799] [ 50.004808] dump_stack_lvl+0x48/0x5e [ 50.004821] warn_alloc+0x162/0x190 [ 50.004832] ? __alloc_pages+0x1fa/0x230 [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 [ 50.004868] vmemmap_populate+0x9e/0x16c [ 50.004878] __populate_section_memmap+0x6a/0xb1 [ 50.004890] section_activate+0x20a/0x278 [ 50.004901] sparse_add_section+0x70/0x160 [ 50.004911] __add_pages+0xc3/0x150 [ 50.004921] add_pages+0x12/0x60 [ 50.004931] add_memory_resource+0x12b/0x320 [ 50.004943] reserve_additional_memory+0x10c/0x150 [ 50.004958] balloon_thread+0x206/0x360 [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 [ 50.004991] kthread+0x16b/0x190 [ 50.005001] ? set_kthread_struct+0x40/0x40 [ 50.005011] ret_from_fork+0x22/0x30 [ 50.005022] Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 After the above, `free` reports correct size (1GB in this case), but that memory seems to be unusable really. "used" is kept low, and soon OOM-killer kicks in. I know the initial 400MB is not much for a full Linux, with X11 etc. But I wouldn't expect it to fail this way when _adding_ memory. I've tried also with initial 800MB. In this case, I do not get "alloc failure" any more, but monitoring `free`, the extra memory still doesn't seem to be used. Any ideas? I can't reproduce that. I started a guest with 8GB of memory, in the guest I'm seeing: # uname -a Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 CET 2022 x86_64 x86_64 x86_64 GNU/Linux # free total used free shared buff/cache available Mem: 817826071628 8023300 8560 83332 8010196 Swap: 20971320 2097132 Then I'm raising the memory for the guest in dom0: # xl list NameID Mem VCPUs State Time(s) Domain-0 0 2634 8 r-1016.5 Xenstore 131 1 -b 0.9 sle15sp1 3 8190 6 -b 184.6 # xl mem-max 3 1 # xenstore-write /local/domain/3/memory/static-max 1024 # xl mem-set 3 1 # xl list NameID Mem VCPUs State Time(s) Domain-0 0 2634 8 r-1018.5 Xenstore 131 1 -b 1.0 sle15sp1 3 1 6 -b 186.7 In the guest I get now: # free total used free shared buff/cache available Mem: 10031700 110904 9734172 8560 186624 9814344 Swap: 20971320 2097132 And after using lots of memory via a ramdisk: # free total used free shared buff/cache available Mem: 10031700 116660 1663840 7181776 8251200 2635372 Swap: 20971320 2097132 You can see buff/cache is now larger than the initial total memory and free is lower than the added memory amount. Juergen OpenPGP_0xB0DE9DD628BF132F.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Increasing domain memory beyond initial maxmem
Hi, I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase domain memory beyond initial maxmem, but I hit few issues. A little context: domains in Qubes OS start with rather little memory (400MB by default) but maxmem set higher (4GB by default). Then, there is qmemman daemon, that adjust balloon targets for domains, based on (among other things) demand reported by the domains themselves. There is also a little swap, to mitigate qmemman latency (few hundreds ms at worst). This initial memory < maxmmem in case of PVH / HVM makes use of PoD which I'm trying to get rid of. But also, IIUC Linux will waste some memory for bookkeeping based on maxmem, not actually usable memory. First issue: after using `xl mem-max`, `xl mem-set` still refuses to increase memory more than initial maxmem. That's because xl mem-max does not update 'memory/static-max' xenstore node. This one is easy to work around. Then, the actual hotplug fails on the domU side with: [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 [ 50.004792] Call Trace: [ 50.004799] [ 50.004808] dump_stack_lvl+0x48/0x5e [ 50.004821] warn_alloc+0x162/0x190 [ 50.004832] ? __alloc_pages+0x1fa/0x230 [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 [ 50.004868] vmemmap_populate+0x9e/0x16c [ 50.004878] __populate_section_memmap+0x6a/0xb1 [ 50.004890] section_activate+0x20a/0x278 [ 50.004901] sparse_add_section+0x70/0x160 [ 50.004911] __add_pages+0xc3/0x150 [ 50.004921] add_pages+0x12/0x60 [ 50.004931] add_memory_resource+0x12b/0x320 [ 50.004943] reserve_additional_memory+0x10c/0x150 [ 50.004958] balloon_thread+0x206/0x360 [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 [ 50.004991] kthread+0x16b/0x190 [ 50.005001] ? set_kthread_struct+0x40/0x40 [ 50.005011] ret_from_fork+0x22/0x30 [ 50.005022] Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 After the above, `free` reports correct size (1GB in this case), but that memory seems to be unusable really. "used" is kept low, and soon OOM-killer kicks in. I know the initial 400MB is not much for a full Linux, with X11 etc. But I wouldn't expect it to fail this way when _adding_ memory. I've tried also with initial 800MB. In this case, I do not get "alloc failure" any more, but monitoring `free`, the extra memory still doesn't seem to be used. Any ideas? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab signature.asc Description: PGP signature
