RE: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-23 Thread Salil Mehta 
> From: Andrew Jones [mailto:drjo...@redhat.com]
> Sent: Tuesday, June 23, 2020 10:12 AM
> To: Salil Mehta 
> Cc: qemu-devel@nongnu.org; qemu-...@nongnu.org; peter.mayd...@linaro.org;
> sudeep.ho...@arm.com; gs...@redhat.com; m...@redhat.com; jiakern...@gmail.com;
> m...@kernel.org; zhukeqian ; da...@redhat.com;
> richard.hender...@linaro.org; Linuxarm ;
> eric.au...@redhat.com; james.mo...@arm.com; catalin.mari...@arm.com;
> imamm...@redhat.com; pbonz...@redhat.com; mehta.salil@gmail.com;
> maran.wil...@oracle.com; w...@kernel.org; wangxiongfeng (C)
> 
> Subject: Re: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> 
> On Sat, Jun 13, 2020 at 10:36:07PM +0100, Salil Mehta wrote:
> > This patch-set introduces the virtual cpu hotplug support for ARMv8
> > architecture in QEMU. Idea is to be able to hotplug and hot-unplug the vcpus
> > while guest VM is running and no reboot is required. This does *not* makes
> any
> > assumption of the physical cpu hotplug availability within the host system
> but
> > rather tries to solve the problem at virtualizer/QEMU layer and by 
> > introducing
> > cpu hotplug hooks and event handling within the guest kernel. No changes are
> > required within the host kernel/KVM.
> >
> > Motivation:
> > This allows scaling the guest VM compute capacity on-demand which would be
> > useful for the following example scenarios,
> > 1. Vertical Pod Autoscaling[3][4] in the cloud: Part of the orchestration
> >framework which could adjust resource requests (CPU and Mem requests) for
> >the containers in a pod, based on usage.
> > 2. Pay-as-you-grow Business Model: Infrastructure provider could allocate 
> > and
> >restrict the total number of compute resources available to the guest VM
> >according to the SLA(Service Level Agreement). VM owner could request for
> >more compute to be hot-plugged for some cost.
> >
> > Terminology:
> >
> > (*) Present cpus: Total cpus with which guest has/will boot and are 
> > available
> >   to guest for use and can be onlined. Qemu parameter(-smp)
> > (*) Disabled cpus: Possible cpus which will not be available for the guest
> to
> >use. These can be hotplugged and made present. These can 
> > be
> >thought of as un-plugged vcpus. These will be included as
> >part of sizing.
> > (*) Posssible cpus: Total vcpus which could ever exist in VM. This includes
> > booted cpus plus any cpus which could be later plugged.
> > - Qemu parameter(-maxcpus)
> > - Possible vcpus = Present vcpus (+) Disabled vcpus
> >
> >
> > Limitations of ARMv8 Architecture:
> >
> > A. Physical Limitation to CPU Hotplug:
> > 1. ARMv8 architecture does not support the concept of the physical cpu 
> > hotplug.
> >The closest thing which is recomended to achieve the cpu hotplug on ARM
> is
> >to bring down power state of the cpu using PSCI.
> > 2. Other ARM components like GIC etc. have not been designed to realize
> >physical cpu hotplug capability as of now.
> >
> > B. Limitations of GIC to Support Virtual CPU Hotplug:
> > 1. GIC requires various resources(related to GICR/redistributor, GICC/cpu
> >interface etc) like memory regions to be fixed at the VM init time and 
> > these
> >could not be changed later on after VM has inited.
> > 2. Associations between GICC(GIC cpu interface) and vcpu get fixed at the VM
> >init time and GIC does not allows to change this association once GIC has
> >initialized.
> >
> > C. Known Limitation of the KVM:
> > 1. As of now KVM allows to create VCPUs but does not allows to delete the
> >already created vcpus. QEMU already provides an interface to manage 
> > created
> >vcpus at KVM level and then to re-use them.
> > 2. Inconsistency in interpretation of the MPIDR generated by KVM for vcpus
> >vis-a-vis SMT/threads. This does not looks to be compliant to the MPIDR
> >format(SMT is present) as mentioned in the ARMv8 spec. (Please correct my
> >understanding if I am wrong here?)
> >
> >
> > Workaround to the problems mentioned in Section B & C1:
> > 1. We pre-size the GIC with possible vcpus at VM init time
> > 2. Pre-create all possible vcpus at KVM and associate them with GICC
> > 3. Park the unplugged vcpus (similar to x86)
> >
> >
> > (*) For all of above please refer to Marc's suggestion here[1]
> >
> >
> > Overview of the Approach:
&

Re: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-23 Thread Andrew Jones 
On Sat, Jun 13, 2020 at 10:36:07PM +0100, Salil Mehta wrote:
> This patch-set introduces the virtual cpu hotplug support for ARMv8
> architecture in QEMU. Idea is to be able to hotplug and hot-unplug the vcpus
> while guest VM is running and no reboot is required. This does *not* makes any
> assumption of the physical cpu hotplug availability within the host system but
> rather tries to solve the problem at virtualizer/QEMU layer and by introducing
> cpu hotplug hooks and event handling within the guest kernel. No changes are
> required within the host kernel/KVM.
> 
> Motivation:
> This allows scaling the guest VM compute capacity on-demand which would be
> useful for the following example scenarios,
> 1. Vertical Pod Autoscaling[3][4] in the cloud: Part of the orchestration
>framework which could adjust resource requests (CPU and Mem requests) for
>the containers in a pod, based on usage.
> 2. Pay-as-you-grow Business Model: Infrastructure provider could allocate and
>restrict the total number of compute resources available to the guest VM
>according to the SLA(Service Level Agreement). VM owner could request for
>more compute to be hot-plugged for some cost.
> 
> Terminology:
> 
> (*) Present cpus: Total cpus with which guest has/will boot and are available
>   to guest for use and can be onlined. Qemu parameter(-smp)
> (*) Disabled cpus: Possible cpus which will not be available for the guest to
>use. These can be hotplugged and made present. These can be
>  thought of as un-plugged vcpus. These will be included as
>  part of sizing.
> (*) Posssible cpus: Total vcpus which could ever exist in VM. This includes
> booted cpus plus any cpus which could be later plugged.
>   - Qemu parameter(-maxcpus)
>   - Possible vcpus = Present vcpus (+) Disabled vcpus
> 
> 
> Limitations of ARMv8 Architecture:
> 
> A. Physical Limitation to CPU Hotplug:
> 1. ARMv8 architecture does not support the concept of the physical cpu 
> hotplug.
>The closest thing which is recomended to achieve the cpu hotplug on ARM is
>to bring down power state of the cpu using PSCI.
> 2. Other ARM components like GIC etc. have not been designed to realize
>physical cpu hotplug capability as of now. 
> 
> B. Limitations of GIC to Support Virtual CPU Hotplug:
> 1. GIC requires various resources(related to GICR/redistributor, GICC/cpu
>interface etc) like memory regions to be fixed at the VM init time and 
> these
>could not be changed later on after VM has inited.
> 2. Associations between GICC(GIC cpu interface) and vcpu get fixed at the VM
>init time and GIC does not allows to change this association once GIC has
>initialized.
> 
> C. Known Limitation of the KVM:
> 1. As of now KVM allows to create VCPUs but does not allows to delete the
>already created vcpus. QEMU already provides an interface to manage created
>vcpus at KVM level and then to re-use them.
> 2. Inconsistency in interpretation of the MPIDR generated by KVM for vcpus
>vis-a-vis SMT/threads. This does not looks to be compliant to the MPIDR
>format(SMT is present) as mentioned in the ARMv8 spec. (Please correct my
>understanding if I am wrong here?)
>
> 
> Workaround to the problems mentioned in Section B & C1:
> 1. We pre-size the GIC with possible vcpus at VM init time
> 2. Pre-create all possible vcpus at KVM and associate them with GICC 
> 3. Park the unplugged vcpus (similar to x86)
> 
> 
> (*) For all of above please refer to Marc's suggestion here[1]
> 
> 
> Overview of the Approach:
> At the time of machvirt_init() we pre-create all of the possible ARMCPU
> objects along with the corresponding KVM vcpus at the host. Disabled KVM vcpu
> (which are *not* "present" vcpus but are part of "possible" vcpu list) are
> parked at per VM list "kvm_parked_vcpus" after their initialization.
> 
> We create the ARMCPU objects(but these are not *realized* in QOM sense) even
> for the disabled vcpus to facilitate the GIC initialization (pre-sized with
> possible vcpus). After Initialization of the machine is complete we release
> the ARMCPU Objects for the disabled vcpus. These ARMCPU object shall be
> re-created at the time when vcpu is hot plugged. This new object is then
> re-attached with the earlier parked KVM vcpu which also gets unparked. The
> ARMCPU object gets now "realized" in QEMU, which means creation of the
> corresponding threads, pre_plug/plug phases, and event notification to the
> guest using ACPI GED etc. Similarly, hot-unplug leg will lead to the
> "unrealization" of the vcpus and will lead to similar ACPI GED events to the
> guest for unplug and cleanup and eventually ARMCPU object shall be released 
> and
> KVM vcpus shall be parked again.
> 
> During machine init, ACPI MADT Table is sized with *possible* vcpus GICC
> entries. The unplugged/disabled vcpus are presented as

RE: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-15 Thread Salil Mehta 
Hi Marc,
Thanks for the review.

> From: Marc Zyngier [mailto:m...@kernel.org]
> Sent: Sunday, June 14, 2020 12:55 PM
> 
> Hi Salil,
> 
> On 2020-06-13 22:36, Salil Mehta wrote:
> > This patch-set introduces the virtual cpu hotplug support for ARMv8
> > architecture in QEMU. Idea is to be able to hotplug and hot-unplug the
> > vcpus
> > while guest VM is running and no reboot is required. This does *not*
> > makes any
> > assumption of the physical cpu hotplug availability within the host
> > system but
> > rather tries to solve the problem at virtualizer/QEMU layer and by
> > introducing
> > cpu hotplug hooks and event handling within the guest kernel. No
> > changes are
> > required within the host kernel/KVM.
> >
> > Motivation:
> > This allows scaling the guest VM compute capacity on-demand which would
> > be
> > useful for the following example scenarios,
> > 1. Vertical Pod Autoscaling[3][4] in the cloud: Part of the
> > orchestration
> >framework which could adjust resource requests (CPU and Mem
> > requests) for
> >the containers in a pod, based on usage.
> > 2. Pay-as-you-grow Business Model: Infrastructure provider could
> > allocate and
> >restrict the total number of compute resources available to the
> > guest VM
> >according to the SLA(Service Level Agreement). VM owner could
> > request for
> >more compute to be hot-plugged for some cost.
> >
> > Terminology:
> >
> > (*) Present cpus: Total cpus with which guest has/will boot and are
> > available
> >   to guest for use and can be onlined. Qemu
> > parameter(-smp)
> > (*) Disabled cpus: Possible cpus which will not be available for the
> > guest to
> >use. These can be hotplugged and made present. These
> > can be
> >thought of as un-plugged vcpus. These will be included as
> >part of sizing.
> > (*) Posssible cpus: Total vcpus which could ever exist in VM. This
> > includes
> > booted cpus plus any cpus which could be later
> > plugged.
> > - Qemu parameter(-maxcpus)
> > - Possible vcpus = Present vcpus (+) Disabled vcpus
> >
> >
> > Limitations of ARMv8 Architecture:
> >
> > A. Physical Limitation to CPU Hotplug:
> > 1. ARMv8 architecture does not support the concept of the physical cpu
> > hotplug.
> >The closest thing which is recomended to achieve the cpu hotplug on
> > ARM is
> >to bring down power state of the cpu using PSCI.
> 
> It isn't so much that the ARMv8 architecture doesn't support CPU
> hotplug. It is that CPU hotplug is largely out of the scope of the ARMv8
> architecture, which is a CPU architecture and not a system architecture.
> Yes, the lack of a comprehensive system architecture is *very* annoying,
> but let's put the blame where it belongs... ;-)


Sure.

 
> > 2. Other ARM components like GIC etc. have not been designed to realize
> >physical cpu hotplug capability as of now.
> >
> > B. Limitations of GIC to Support Virtual CPU Hotplug:
> > 1. GIC requires various resources(related to GICR/redistributor,
> > GICC/cpu
> >interface etc) like memory regions to be fixed at the VM init time
> > and these
> >could not be changed later on after VM has inited.
> > 2. Associations between GICC(GIC cpu interface) and vcpu get fixed at
> > the VM
> >init time and GIC does not allows to change this association once
> > GIC has
> >initialized.
> 
> There isn't an association, really. the GIC CPU interface is part of the
> CPU itself, and not an external entity. KVM doesn't split the two
> either. It is the association between the CPU and its redistributor that
> is being done. There is no architectural way to set this up this, so KVM
> just statically configures these based on the number of vcpus and the
> number/size of redistributor ranges.


I stand corrected. Sorry for the horrible mix up and I realized that some
how I copied the same terminology at other 2 other places as well, maybe
under sleep duress :(. I will correct it in later versions.

To be frank, I actually meant association of "mp-affinity" and the
"proc number" as given by GICR_TYPER register for the vgic. I guess reading
this register using kvm_gicr_acces/KVM_DEV_ARM_VGIC_GRP_REDIST_REGS from
QEMU lands up in vgic vgic_mmio_read_v3r_typer() which forms the reg value
using "vcpu-id" and "mpidr"(fetched from MPIDR_EL1).

Also, value of the MPIDR for vcpu is set during KVM_ARM_VCPU_INIT IOCTL
from QEMU after the creation of the vcpus(using KVM_CREATE_VCPU). I guess
this is done during reset of all of the system regs  SYS_MPIDR_EL1 value
is also reset to default within function reset_mpidr() derived using below
logic:

   +++++++++
  MPIDR   |||  Res   |   Aff2  |   Aff1   |  Aff0|
   +++++++++
\  \  \   | |
 \

Re: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-14 Thread Marc Zyngier 

Hi Salil,

On 2020-06-13 22:36, Salil Mehta wrote:

This patch-set introduces the virtual cpu hotplug support for ARMv8
architecture in QEMU. Idea is to be able to hotplug and hot-unplug the 
vcpus
while guest VM is running and no reboot is required. This does *not* 
makes any
assumption of the physical cpu hotplug availability within the host 
system but
rather tries to solve the problem at virtualizer/QEMU layer and by 
introducing
cpu hotplug hooks and event handling within the guest kernel. No 
changes are

required within the host kernel/KVM.

Motivation:
This allows scaling the guest VM compute capacity on-demand which would 
be

useful for the following example scenarios,
1. Vertical Pod Autoscaling[3][4] in the cloud: Part of the 
orchestration
   framework which could adjust resource requests (CPU and Mem 
requests) for

   the containers in a pod, based on usage.
2. Pay-as-you-grow Business Model: Infrastructure provider could 
allocate and
   restrict the total number of compute resources available to the 
guest VM
   according to the SLA(Service Level Agreement). VM owner could 
request for

   more compute to be hot-plugged for some cost.

Terminology:

(*) Present cpus: Total cpus with which guest has/will boot and are 
available
  to guest for use and can be onlined. Qemu 
parameter(-smp)
(*) Disabled cpus: Possible cpus which will not be available for the 
guest to
   use. These can be hotplugged and made present. These 
can be

   thought of as un-plugged vcpus. These will be included as
   part of sizing.
(*) Posssible cpus: Total vcpus which could ever exist in VM. This 
includes
booted cpus plus any cpus which could be later 
plugged.

- Qemu parameter(-maxcpus)
- Possible vcpus = Present vcpus (+) Disabled vcpus


Limitations of ARMv8 Architecture:

A. Physical Limitation to CPU Hotplug:
1. ARMv8 architecture does not support the concept of the physical cpu 
hotplug.
   The closest thing which is recomended to achieve the cpu hotplug on 
ARM is

   to bring down power state of the cpu using PSCI.


It isn't so much that the ARMv8 architecture doesn't support CPU 
hotplug. It is that CPU hotplug is largely out of the scope of the ARMv8 
architecture, which is a CPU architecture and not a system architecture. 
Yes, the lack of a comprehensive system architecture is *very* annoying, 
but let's put the blame where it belongs... ;-)



2. Other ARM components like GIC etc. have not been designed to realize
   physical cpu hotplug capability as of now.

B. Limitations of GIC to Support Virtual CPU Hotplug:
1. GIC requires various resources(related to GICR/redistributor, 
GICC/cpu
   interface etc) like memory regions to be fixed at the VM init time 
and these

   could not be changed later on after VM has inited.
2. Associations between GICC(GIC cpu interface) and vcpu get fixed at 
the VM
   init time and GIC does not allows to change this association once 
GIC has

   initialized.


There isn't an association, really. the GIC CPU interface is part of the 
CPU itself, and not an external entity. KVM doesn't split the two 
either. It is the association between the CPU and its redistributor that 
is being done. There is no architectural way to set this up this, so KVM 
just statically configures these based on the number of vcpus and the 
number/size of redistributor ranges.




C. Known Limitation of the KVM:
1. As of now KVM allows to create VCPUs but does not allows to delete 
the
   already created vcpus. QEMU already provides an interface to manage 
created

   vcpus at KVM level and then to re-use them.
2. Inconsistency in interpretation of the MPIDR generated by KVM for 
vcpus
   vis-a-vis SMT/threads. This does not looks to be compliant to the 
MPIDR
   format(SMT is present) as mentioned in the ARMv8 spec. (Please 
correct my

   understanding if I am wrong here?)


I'm unsure of which part of the architecture KVM doesn't follow when 
assigning a MPIDR to a vcpu. By having MPIDR_EL1.MT to 0, KVM tells the 
guest OS that vcpus having the same Aff3-1 fields are mostly 
independent.


This is because:
- we make no placement guarantee whatsoever (this is userspace's 
business)
- ARMv8.2 CPUs designed by ARM (as opposed to architecture licensees) 
always set this bit to 1 as they can hypothetically be coupled to SMT 
CPUs in a big-little system. This makes MT totally meaningless.
- the one SMT implementation available in the wild (ThunderX-2) is 
broken enough that you really should consider turning SMT off (see 
CAVIUM_TX2_ERRATUM_219)
- KVM actively prevents the sharing of resources such as TLBs across 
vcpus


Given the above, I fail to see the point in setting the SMT bit to 
anything but 0.





Workaround to the problems mentioned in Section B & C1:
1. We pre-size the GIC with possible vcpus at VM init time
2. Pre-create all possible vcpus at KVM and

Re: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-13 Thread no-reply 
Patchew URL: 
https://patchew.org/QEMU/20200613213629.21984-1-salil.me...@huawei.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200613213629.21984-1-salil.me...@huawei.com
Subject: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
dc56c53 arm/cpuhp: Add support of *unrealize* ARMCPU during vcpu hot-unplug
988141e arm/cpuhp: Changes required for reset and to support next boot
d5feac6 arm/cpuhp: Update the guest(via GED) about cpu hot-(un)plug events
185a83e arm/cpuhp: Changes required to (re)init the vcpu register info
659f80f arm/cpuhp: Changes to update GIC with vcpu hot-plug notification
bcfc8e7 arm/cpuhp: Changes to (un)wire GICC<->VCPU IRQs during hot-(un)plug
643af9e arm/cpuhp: Add/update basic hot-(un)plug framework
8e17131 arm/cpuhp: Update ACPI GED framework to support vcpu hotplug
a8d5f8e arm/cpuhp: Release objects for *disabled* possible vcpus after init
1f520fd arm/cpuhp: Add ACPI _MAT entry for Processor object
e9543c1 arm/cpuhp: MADT Tbl change to size the guest with possible vcpus
a41164f arm/cpuhp: Update GED _EVT method AML with cpu scan
72b8a98 arm/cpuhp: Update CPUs AML with cpu-(ctrl)dev change
9d81444 arm/cpuhp: Init GED framework with cpu hotplug events
96ef87b arm/cpuhp: Enable ACPI support for vcpu hotplug
be81157 arm/cpuhp: Init PMU at host for all possible vcpus
cb588d9 arm/cpuhp: Changes to pre-size GIC with possible vcpus @machine init
75fe8dd arm/cpuhp: Pre-create disabled possible vcpus @machine init
55adeca arm/cpuhp: Machine init time change common to vcpu {cold|hot}-plug
9c53779 arm/cpuhp: Add common cpu utility for possible vcpus
36a71f4 arm/cpuhp: Add new ARMCPU core-id property
ecab7ce arm/cpuhp: Add QMP vcpu params validation support

=== OUTPUT BEGIN ===
1/22 Checking commit ecab7ce0dc60 (arm/cpuhp: Add QMP vcpu params validation 
support)
2/22 Checking commit 36a71f45080c (arm/cpuhp: Add new ARMCPU core-id property)
3/22 Checking commit 9c53779a7878 (arm/cpuhp: Add common cpu utility for 
possible vcpus)
ERROR: return is not a function, parentheses are not required
#51: FILE: cpus-common.c:104:
+return (cpu && !cpu->disabled);

total: 1 errors, 0 warnings, 65 lines checked

Patch 3/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

4/22 Checking commit 55adeca4f0c2 (arm/cpuhp: Machine init time change common 
to vcpu {cold|hot}-plug)
ERROR: space prohibited between function name and open parenthesis '('
#206: FILE: hw/arm/virt.c:2143:
+assert (found_cpu);

ERROR: space required before the open brace '{'
#341: FILE: hw/arm/virt.c:2338:
+   (object_dynamic_cast(OBJECT(dev), TYPE_CPU))){

total: 2 errors, 0 warnings, 375 lines checked

Patch 4/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

5/22 Checking commit 75fe8dd750d3 (arm/cpuhp: Pre-create disabled possible 
vcpus @machine init)
ERROR: space required after that ',' (ctx:VxV)
#93: FILE: hw/arm/virt.c:1841:
+qdev_set_id(DEVICE(cpuobj),core_id);
   ^

ERROR: suspect code indent for conditional statements (12, 15)
#109: FILE: hw/arm/virt.c:1857:
+if (kvm_enabled()) {
+   kvm_arm_create_host_vcpu(ARM_CPU(cs));

WARNING: line over 80 characters
#113: FILE: hw/arm/virt.c:1861:
+ * Add disabled vcpu to cpu slot during the init phase of the virt 
machine.

WARNING: line over 80 characters
#119: FILE: hw/arm/virt.c:1867:
+ * 2. Now, after initialization of the virt machine is complete we 
could use

WARNING: line over 80 characters
#127: FILE: hw/arm/virt.c:1875:
+ *We will use the (ii) approach and release the ARMCPU objects 
after GIC

total: 2 errors, 3 warnings, 159 lines checked

Patch 5/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

6/22 Checking commit cb588d911d62 (arm/cpuhp: Changes to pre-size GIC with 
possible vcpus @machine init)
ERROR: braces {} are necessary for all arms of this statement
#105: FILE: hw/intc/arm_gicv3_common.c:355:
+if (qemu_present_cpu(cpu))
[...]
+else
[...]

ERROR: braces {} are necessary for all arms of this statement
#121: FILE: hw/intc/arm_gicv3_cpuif.c:782:
+if (!qemu_present_cpu(cs->cpu))
[...]

ERROR: braces {} are necessary for all arms of this statement
#161: FILE: hw/i

Re: [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-13 Thread no-reply 
Patchew URL: 
https://patchew.org/QEMU/20200613213629.21984-1-salil.me...@huawei.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  TESTcheck-unit: tests/test-char
  TESTcheck-qtest-aarch64: tests/qtest/tpm-tis-device-swtpm-test
  TESTcheck-qtest-aarch64: tests/qtest/numa-test
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(8) with smp-cpus(8)
  TESTiotest-qcow2: 003
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(8) with smp-cpus(8)
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(8) with smp-cpus(8)
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(8) with smp-cpus(8)
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(2) with smp-cpus(2)
  TESTcheck-qtest-aarch64: tests/qtest/boot-serial-test
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(1) with smp-cpus(1)
  TESTcheck-qtest-aarch64: tests/qtest/migration-test
  TESTcheck-unit: tests/check-qnum
  TESTiotest-qcow2: 004
---
qemu-system-aarch64: falling back to tcg
  TESTiotest-qcow2: 060
  TESTcheck-qtest-aarch64: tests/qtest/bios-tables-test
qemu-system-aarch64: warning: For GICv2 max-cpus must be equal to smp-cpus
qemu-system-aarch64: warning: Overriding specified max-cpus(1) with smp-cpus(1)
  TESTiotest-qcow2: 061
  TESTiotest-qcow2: 062
  TESTiotest-qcow2: 063
---
acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-3PFIM0], 
Expected [aml:tests/data/acpi/virt/DSDT].
See source file tests/qtest/bios-tables-test.c for instructions on how to 
update expected files.
to see ASL diff between mismatched files install IASL, rebuild QEMU from 
scratch and re-run tests with V=1 environment variable set**
ERROR:/tmp/qemu-test/src/tests/qtest/bios-tables-test.c:494:test_acpi_asl: 
assertion failed: (all_tables_match)
ERROR - Bail out! 
ERROR:/tmp/qemu-test/src/tests/qtest/bios-tables-test.c:494:test_acpi_asl: 
assertion failed: (all_tables_match)
make: *** [check-qtest-aarch64] Error 1
make: *** Waiting for unfinished jobs
  TESTiotest-qcow2: 073
  TESTiotest-qcow2: 074
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', 
'--label', 'com.qemu.instance.uuid=85b55b455d55449297469bf3a981cabd', '-u', 
'1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', 
'-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 
'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', 
'/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', 
'/var/tmp/patchew-tester-tmp-i8pzxavb/src/docker-src.2020-06-13-18.08.03.29364:/var/tmp/qemu:z,ro',
 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit 
status 2.
filter=--filter=label=com.qemu.instance.uuid=85b55b455d55449297469bf3a981cabd
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-i8pzxavb/src'
make: *** [docker-run-test-quick@centos7] Error 2

real16m30.871s
user0m9.410s


The full log is available at
http://patchew.org/logs/20200613213629.21984-1-salil.me...@huawei.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch

2020-06-13 Thread Salil Mehta 
This patch-set introduces the virtual cpu hotplug support for ARMv8
architecture in QEMU. Idea is to be able to hotplug and hot-unplug the vcpus
while guest VM is running and no reboot is required. This does *not* makes any
assumption of the physical cpu hotplug availability within the host system but
rather tries to solve the problem at virtualizer/QEMU layer and by introducing
cpu hotplug hooks and event handling within the guest kernel. No changes are
required within the host kernel/KVM.

Motivation:
This allows scaling the guest VM compute capacity on-demand which would be
useful for the following example scenarios,
1. Vertical Pod Autoscaling[3][4] in the cloud: Part of the orchestration
   framework which could adjust resource requests (CPU and Mem requests) for
   the containers in a pod, based on usage.
2. Pay-as-you-grow Business Model: Infrastructure provider could allocate and
   restrict the total number of compute resources available to the guest VM
   according to the SLA(Service Level Agreement). VM owner could request for
   more compute to be hot-plugged for some cost.

Terminology:

(*) Present cpus: Total cpus with which guest has/will boot and are available
  to guest for use and can be onlined. Qemu parameter(-smp)
(*) Disabled cpus: Possible cpus which will not be available for the guest to
   use. These can be hotplugged and made present. These can be
   thought of as un-plugged vcpus. These will be included as
   part of sizing.
(*) Posssible cpus: Total vcpus which could ever exist in VM. This includes
booted cpus plus any cpus which could be later plugged.
- Qemu parameter(-maxcpus)
- Possible vcpus = Present vcpus (+) Disabled vcpus


Limitations of ARMv8 Architecture:

A. Physical Limitation to CPU Hotplug:
1. ARMv8 architecture does not support the concept of the physical cpu hotplug.
   The closest thing which is recomended to achieve the cpu hotplug on ARM is
   to bring down power state of the cpu using PSCI.
2. Other ARM components like GIC etc. have not been designed to realize
   physical cpu hotplug capability as of now. 

B. Limitations of GIC to Support Virtual CPU Hotplug:
1. GIC requires various resources(related to GICR/redistributor, GICC/cpu
   interface etc) like memory regions to be fixed at the VM init time and these
   could not be changed later on after VM has inited.
2. Associations between GICC(GIC cpu interface) and vcpu get fixed at the VM
   init time and GIC does not allows to change this association once GIC has
   initialized.

C. Known Limitation of the KVM:
1. As of now KVM allows to create VCPUs but does not allows to delete the
   already created vcpus. QEMU already provides an interface to manage created
   vcpus at KVM level and then to re-use them.
2. Inconsistency in interpretation of the MPIDR generated by KVM for vcpus
   vis-a-vis SMT/threads. This does not looks to be compliant to the MPIDR
   format(SMT is present) as mentioned in the ARMv8 spec. (Please correct my
   understanding if I am wrong here?)
   

Workaround to the problems mentioned in Section B & C1:
1. We pre-size the GIC with possible vcpus at VM init time
2. Pre-create all possible vcpus at KVM and associate them with GICC 
3. Park the unplugged vcpus (similar to x86)


(*) For all of above please refer to Marc's suggestion here[1]


Overview of the Approach:
At the time of machvirt_init() we pre-create all of the possible ARMCPU
objects along with the corresponding KVM vcpus at the host. Disabled KVM vcpu
(which are *not* "present" vcpus but are part of "possible" vcpu list) are
parked at per VM list "kvm_parked_vcpus" after their initialization.

We create the ARMCPU objects(but these are not *realized* in QOM sense) even
for the disabled vcpus to facilitate the GIC initialization (pre-sized with
possible vcpus). After Initialization of the machine is complete we release
the ARMCPU Objects for the disabled vcpus. These ARMCPU object shall be
re-created at the time when vcpu is hot plugged. This new object is then
re-attached with the earlier parked KVM vcpu which also gets unparked. The
ARMCPU object gets now "realized" in QEMU, which means creation of the
corresponding threads, pre_plug/plug phases, and event notification to the
guest using ACPI GED etc. Similarly, hot-unplug leg will lead to the
"unrealization" of the vcpus and will lead to similar ACPI GED events to the
guest for unplug and cleanup and eventually ARMCPU object shall be released and
KVM vcpus shall be parked again.

During machine init, ACPI MADT Table is sized with *possible* vcpus GICC
entries. The unplugged/disabled vcpus are presented as MADT GICC DISABLED
entries to the guest. This means the guest will have its resources pre-sized
with possible vcpus(=present+disabled)

Other approaches to deal with ARMCPU object release(after machine init):
1. The ARMCPU objects for the