Re: [Qemu-ppc] KVM memory slots limit on powerpc
On 04/09/15 12:04, Alexander Graf wrote:
>
>
> On 04.09.15 11:59, Christian Borntraeger wrote:
>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>
>>> Hi all,
>>>
>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>> you'll see that it aborts way earlier already.
>>>
>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>
>> When you are at it, the s390 value should also be increased I guess.
>
> That constant defines the array size for the memslot array in struct kvm
> which in turn again gets allocated by kzalloc, so it's pinned kernel
> memory that is physically contiguous. Doing big allocations can turn
> into problems during runtime.
FWIW, I've just checked sizeof(struct kvm) with the current ppc64 kernel
build from master branch, and it is 34144 bytes.
So on a system that is using PAGE_SIZE = 64kB, there should be plenty of
space left before we're getting into trouble.
And even assuming the worst case, that we're on a system which still
uses PAGE_SIZE = 4kB, the last page of the 34144 bytes is only filled
with 1376 bytes, leaving 2720 bytes free right now.
sizeof(struct kvm_memory_slot) is 48 bytes right now on powerpc, and you
need two additional bytes per entry for the id_to_index array in
struct kvm_memslots, i.e. we need 50 additional bytes per entry on ppc.
That means we could increase KVM_USER_MEM_SLOTS by 2720 / 50 = 54
entries without getting into further trouble.
I think we should leave some more additional bytes left in that last
4k page of the struct kvm region, ... so what about increasing
KVM_USER_MEM_SLOTS to 32 + 48 = 80 now (instead of 32 + 54 = 86) to
ease the memslot situation at least a little bit 'till we figured out
a really final solution like growable memslots?
Thomas
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-ppc] KVM memory slots limit on powerpc
On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote:
> On 04/09/15 12:07, Christian Borntraeger wrote:
> > Am 04.09.2015 um 12:04 schrieb Alexander Graf:
> >>
> >>
> >> On 04.09.15 11:59, Christian Borntraeger wrote:
> >>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>
> Hi all,
>
> now that we get memory hotplugging for the spapr machine on qemu-ppc,
> too, it seems like we easily can hit the amount of KVM-internal memory
> slots now ("#define KVM_USER_MEM_SLOTS 32" in
> arch/powerpc/include/asm/kvm_host.h). For example, start
> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
> you'll see that it aborts way earlier already.
>
> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
> already (+3 internal slots = 512) ... maybe we should now increase the
> amount of slots on powerpc, too? Since we don't use internal slots on
> POWER, would 512 be a good value? Or would less be sufficient, too?
> >>>
> >>> When you are at it, the s390 value should also be increased I guess.
> >>
> >> That constant defines the array size for the memslot array in struct kvm
> >> which in turn again gets allocated by kzalloc, so it's pinned kernel
> >> memory that is physically contiguous. Doing big allocations can turn
> >> into problems during runtime.
> >>
> >> So maybe there is another way? Can we extend the memslot array size
> >> dynamically somehow? Allocate it separately? How much memory does the
> >> memslot array use up with 512 entries?
> >
> > Maybe some rcu protected scheme that doubles the amount of memslots for
> > each overrun? Yes, that would be good and even reduce the footprint for
> > systems with only a small number of memslots.
>
> Seems like Alex Williamson already posted a patchset for growable
> memslots a couple of years ago:
>
> http://www.spinics.net/lists/kvm/msg50491.html
>
> But I didn't quite spot the result in that thread why it never has been
> included upstream. Alex (W.), do you remember the outcome?
IIRC it was simply thought to be more complicated than necessary. Once
we started caching memory slot misses, the O(N) search of a larger fixed
array didn't seem to bother anyone, so I abandoned the weight-balanced
tree, which I think still had some re-balancing issues. Please run with
the code and make it work if it's useful now. Thanks,
Alex
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] powerpc/e500: move qemu machine spec together with the rest
This way we get rid of an entire file with mostly
duplicated code plus a Kconfig option that you always
had to take care to check it in order for kvm to work.
Signed-off-by: Laurentiu Tudor
---
arch/powerpc/platforms/85xx/Kconfig | 15 -
arch/powerpc/platforms/85xx/Makefile | 1 -
arch/powerpc/platforms/85xx/corenet_generic.c | 1 +
arch/powerpc/platforms/85xx/qemu_e500.c | 85 ---
4 files changed, 1 insertion(+), 101 deletions(-)
delete mode 100644 arch/powerpc/platforms/85xx/qemu_e500.c
diff --git a/arch/powerpc/platforms/85xx/Kconfig
b/arch/powerpc/platforms/85xx/Kconfig
index 97915fe..0c1ce10 100644
--- a/arch/powerpc/platforms/85xx/Kconfig
+++ b/arch/powerpc/platforms/85xx/Kconfig
@@ -249,21 +249,6 @@ config MVME2500
endif # PPC32
-config PPC_QEMU_E500
- bool "QEMU generic e500 platform"
- select DEFAULT_UIMAGE
- help
- This option enables support for running as a QEMU guest using
- QEMU's generic e500 machine. This is not required if you're
- using a QEMU machine that targets a specific board, such as
- mpc8544ds.
-
- Unlike most e500 boards that target a specific CPU, this
- platform works with any e500-family CPU that QEMU supports.
- Thus, you'll need to make sure CONFIG_PPC_E500MC is set or
- unset based on the emulated CPU (or actual host CPU in the case
- of KVM).
-
config CORENET_GENERIC
bool "Freescale CoreNet Generic"
select DEFAULT_UIMAGE
diff --git a/arch/powerpc/platforms/85xx/Makefile
b/arch/powerpc/platforms/85xx/Makefile
index 1fe7fb9..33fbfb8 100644
--- a/arch/powerpc/platforms/85xx/Makefile
+++ b/arch/powerpc/platforms/85xx/Makefile
@@ -29,6 +29,5 @@ obj-$(CONFIG_SOCRATES)+= socrates.o socrates_fpga_pic.o
obj-$(CONFIG_KSI8560)+= ksi8560.o
obj-$(CONFIG_XES_MPC85xx) += xes_mpc85xx.o
obj-$(CONFIG_GE_IMP3A) += ge_imp3a.o
-obj-$(CONFIG_PPC_QEMU_E500) += qemu_e500.o
obj-$(CONFIG_SGY_CTS1000) += sgy_cts1000.o
obj-$(CONFIG_MVME2500) += mvme2500.o
diff --git a/arch/powerpc/platforms/85xx/corenet_generic.c
b/arch/powerpc/platforms/85xx/corenet_generic.c
index bd839dc..85078ee 100644
--- a/arch/powerpc/platforms/85xx/corenet_generic.c
+++ b/arch/powerpc/platforms/85xx/corenet_generic.c
@@ -159,6 +159,7 @@ static const char * const boards[] __initconst = {
"fsl,T1042RDB",
"fsl,T1042RDB_PI",
"keymile,kmcoge4",
+ "fsl,qemu-e500",
NULL
};
diff --git a/arch/powerpc/platforms/85xx/qemu_e500.c
b/arch/powerpc/platforms/85xx/qemu_e500.c
deleted file mode 100644
index 8ad2fe6..000
--- a/arch/powerpc/platforms/85xx/qemu_e500.c
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Paravirt target for a generic QEMU e500 machine
- *
- * This is intended to be a flexible device-tree-driven platform, not fixed
- * to a particular piece of hardware or a particular spec of virtual hardware,
- * beyond the assumption of an e500-family CPU. Some things are still
hardcoded
- * here, such as MPIC, but this is a limitation of the current code rather than
- * an interface contract with QEMU.
- *
- * Copyright 2012 Freescale Semiconductor Inc.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
-
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include "smp.h"
-#include "mpc85xx.h"
-
-void __init qemu_e500_pic_init(void)
-{
- struct mpic *mpic;
- unsigned int flags = MPIC_BIG_ENDIAN | MPIC_SINGLE_DEST_CPU |
- MPIC_ENABLE_COREINT;
-
- mpic = mpic_alloc(NULL, 0, flags, 0, 256, " OpenPIC ");
-
- BUG_ON(mpic == NULL);
- mpic_init(mpic);
-}
-
-static void __init qemu_e500_setup_arch(void)
-{
- ppc_md.progress("qemu_e500_setup_arch()", 0);
-
- fsl_pci_assign_primary();
- swiotlb_detect_4g();
-#if defined(CONFIG_FSL_PCI) && defined(CONFIG_ZONE_DMA32)
- /*
-* Inbound windows don't cover the full lower 4 GiB
-* due to conflicts with PCICSRBAR and outbound windows,
-* so limit the DMA32 zone to 2 GiB, to allow consistent
-* allocations to succeed.
-*/
- limit_zone_pfn(ZONE_DMA32, 1UL << (31 - PAGE_SHIFT));
-#endif
- mpc85xx_smp_init();
-}
-
-/*
- * Called very early, device-tree isn't unflattened
- */
-static int __init qemu_e500_probe(void)
-{
- unsigned long root = of_get_flat_dt_root();
-
- return !!of_flat_dt_is_compatible(root, "fsl,qemu-e500");
-}
-
-machine_arch_initcall(qemu_e500, mpc85xx_common_publish_devices);
-
-define_machine(qemu_e500) {
- .name = "QEMU e500",
- .probe = qemu_e500_probe,
- .setup_arch = qemu_e500_setup_arch,
-
Re: [Qemu-ppc] KVM memory slots limit on powerpc
On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote: > > Maybe some rcu protected scheme that doubles the amount of memslots > > for > > each overrun? Yes, that would be good and even reduce the footprint > > for > > systems with only a small number of memslots. > > Seems like Alex Williamson already posted a patchset for growable > memslots a couple of years ago: > > http://www.spinics.net/lists/kvm/msg50491.html > > But I didn't quite spot the result in that thread why it never has > been > included upstream. Alex (W.), do you remember the outcome? Isn't the memslot array *already* protected by RCU anyway ? Cheers, Ben. -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-ppc] KVM memory slots limit on powerpc
On 04/09/15 12:07, Christian Borntraeger wrote:
> Am 04.09.2015 um 12:04 schrieb Alexander Graf:
>>
>>
>> On 04.09.15 11:59, Christian Borntraeger wrote:
>>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
Hi all,
now that we get memory hotplugging for the spapr machine on qemu-ppc,
too, it seems like we easily can hit the amount of KVM-internal memory
slots now ("#define KVM_USER_MEM_SLOTS 32" in
arch/powerpc/include/asm/kvm_host.h). For example, start
qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
you'll see that it aborts way earlier already.
The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
already (+3 internal slots = 512) ... maybe we should now increase the
amount of slots on powerpc, too? Since we don't use internal slots on
POWER, would 512 be a good value? Or would less be sufficient, too?
>>>
>>> When you are at it, the s390 value should also be increased I guess.
>>
>> That constant defines the array size for the memslot array in struct kvm
>> which in turn again gets allocated by kzalloc, so it's pinned kernel
>> memory that is physically contiguous. Doing big allocations can turn
>> into problems during runtime.
>>
>> So maybe there is another way? Can we extend the memslot array size
>> dynamically somehow? Allocate it separately? How much memory does the
>> memslot array use up with 512 entries?
>
> Maybe some rcu protected scheme that doubles the amount of memslots for
> each overrun? Yes, that would be good and even reduce the footprint for
> systems with only a small number of memslots.
Seems like Alex Williamson already posted a patchset for growable
memslots a couple of years ago:
http://www.spinics.net/lists/kvm/msg50491.html
But I didn't quite spot the result in that thread why it never has been
included upstream. Alex (W.), do you remember the outcome?
Thomas
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-ppc] KVM memory slots limit on powerpc
Am 04.09.2015 um 12:04 schrieb Alexander Graf:
>
>
> On 04.09.15 11:59, Christian Borntraeger wrote:
>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>
>>> Hi all,
>>>
>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>> you'll see that it aborts way earlier already.
>>>
>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>
>> When you are at it, the s390 value should also be increased I guess.
>
> That constant defines the array size for the memslot array in struct kvm
> which in turn again gets allocated by kzalloc, so it's pinned kernel
> memory that is physically contiguous. Doing big allocations can turn
> into problems during runtime.
>
> So maybe there is another way? Can we extend the memslot array size
> dynamically somehow? Allocate it separately? How much memory does the
> memslot array use up with 512 entries?
Maybe some rcu protected scheme that doubles the amount of memslots for
each overrun? Yes, that would be good and even reduce the footprint for
systems with only a small number of memslots.
Christian
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-ppc] KVM memory slots limit on powerpc
On 04.09.15 11:59, Christian Borntraeger wrote:
> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>
>> Hi all,
>>
>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>> too, it seems like we easily can hit the amount of KVM-internal memory
>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>> arch/powerpc/include/asm/kvm_host.h). For example, start
>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>> you'll see that it aborts way earlier already.
>>
>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>> already (+3 internal slots = 512) ... maybe we should now increase the
>> amount of slots on powerpc, too? Since we don't use internal slots on
>> POWER, would 512 be a good value? Or would less be sufficient, too?
>
> When you are at it, the s390 value should also be increased I guess.
That constant defines the array size for the memslot array in struct kvm
which in turn again gets allocated by kzalloc, so it's pinned kernel
memory that is physically contiguous. Doing big allocations can turn
into problems during runtime.
So maybe there is another way? Can we extend the memslot array size
dynamically somehow? Allocate it separately? How much memory does the
memslot array use up with 512 entries?
Alex
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM memory slots limit on powerpc
Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>
> Hi all,
>
> now that we get memory hotplugging for the spapr machine on qemu-ppc,
> too, it seems like we easily can hit the amount of KVM-internal memory
> slots now ("#define KVM_USER_MEM_SLOTS 32" in
> arch/powerpc/include/asm/kvm_host.h). For example, start
> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
> you'll see that it aborts way earlier already.
>
> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
> already (+3 internal slots = 512) ... maybe we should now increase the
> amount of slots on powerpc, too? Since we don't use internal slots on
> POWER, would 512 be a good value? Or would less be sufficient, too?
When you are at it, the s390 value should also be increased I guess.
Christian
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
KVM memory slots limit on powerpc
Hi all,
now that we get memory hotplugging for the spapr machine on qemu-ppc,
too, it seems like we easily can hit the amount of KVM-internal memory
slots now ("#define KVM_USER_MEM_SLOTS 32" in
arch/powerpc/include/asm/kvm_host.h). For example, start
qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
you'll see that it aborts way earlier already.
The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
already (+3 internal slots = 512) ... maybe we should now increase the
amount of slots on powerpc, too? Since we don't use internal slots on
POWER, would 512 be a good value? Or would less be sufficient, too?
Thomas
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
