date:20200915

Re: [PATCH] hw/arm/aspeed: Map the UART5 device unconditionally

2020-09-15 Thread Cédric Le Goater

On 9/15/20 7:23 PM, Philippe Mathieu-Daudé wrote:
> ping?

It's reviewed : 

  
http://patchwork.ozlabs.org/project/qemu-devel/patch/20200905212415.760452-1-f4...@amsat.org/

I will send a PR when I have more patches.

Thanks,

C. 

> On 9/5/20 11:24 PM, Philippe Mathieu-Daudé wrote:
>> The UART5 is present on the machine regardless there is a
>> character device connected to it. Map it unconditionally.
>>
>> Signed-off-by: Philippe Mathieu-Daudé 
>> ---
>>  hw/arm/aspeed_ast2600.c | 8 +++-
>>  hw/arm/aspeed_soc.c | 8 +++-
>>  2 files changed, 6 insertions(+), 10 deletions(-)
>>
>> diff --git a/hw/arm/aspeed_ast2600.c b/hw/arm/aspeed_ast2600.c
>> index 9d95e421435..1450bde7cf2 100644
>> --- a/hw/arm/aspeed_ast2600.c
>> +++ b/hw/arm/aspeed_ast2600.c
>> @@ -325,11 +325,9 @@ static void aspeed_soc_ast2600_realize(DeviceState 
>> *dev, Error **errp)
>>  }
>>  
>>  /* UART - attach an 8250 to the IO space as our UART5 */
>> -if (serial_hd(0)) {
>> -qemu_irq uart5 = aspeed_soc_get_irq(s, ASPEED_DEV_UART5);
>> -serial_mm_init(get_system_memory(), sc->memmap[ASPEED_DEV_UART5], 2,
>> -   uart5, 38400, serial_hd(0), DEVICE_LITTLE_ENDIAN);
>> -}
>> +serial_mm_init(get_system_memory(), sc->memmap[ASPEED_DEV_UART5], 2,
>> +   aspeed_soc_get_irq(s, ASPEED_DEV_UART5),
>> +   38400, serial_hd(0), DEVICE_LITTLE_ENDIAN);
>>  
>>  /* I2C */
>>  object_property_set_link(OBJECT(>i2c), "dram", OBJECT(s->dram_mr),
>> diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
>> index 35be126db6f..7eefd54ac07 100644
>> --- a/hw/arm/aspeed_soc.c
>> +++ b/hw/arm/aspeed_soc.c
>> @@ -283,11 +283,9 @@ static void aspeed_soc_realize(DeviceState *dev, Error 
>> **errp)
>>  }
>>  
>>  /* UART - attach an 8250 to the IO space as our UART5 */
>> -if (serial_hd(0)) {
>> -qemu_irq uart5 = aspeed_soc_get_irq(s, ASPEED_DEV_UART5);
>> -serial_mm_init(get_system_memory(), sc->memmap[ASPEED_DEV_UART5], 2,
>> -   uart5, 38400, serial_hd(0), DEVICE_LITTLE_ENDIAN);
>> -}
>> +serial_mm_init(get_system_memory(), sc->memmap[ASPEED_DEV_UART5], 2,
>> +   aspeed_soc_get_irq(s, ASPEED_DEV_UART5), 38400,
>> +   serial_hd(0), DEVICE_LITTLE_ENDIAN);
>>  
>>  /* I2C */
>>  object_property_set_link(OBJECT(>i2c), "dram", OBJECT(s->dram_mr),
>>

[Bug 1894869] Re: Chelsio T4 has old MSIX PBA offset bug

2020-09-15 Thread Alex Williamson

The device ID on function 7 is 0x which is typically not valid,
there's no entry for it in the PCI IDs database.  Someone from Chelsio
would need to explain why it's even exposed, but there doesn't seem to
be any value in quirking it since it provides no useful function.

** Changed in: qemu
   Status: New => Invalid

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894869

Title:
  Chelsio T4 has old MSIX PBA offset bug

Status in QEMU:
  Invalid
Status in Debian:
  In Progress

Bug description:
  There exists a bug with Chelsio NICs T4 that causes the following
  error:

  kvm: -device vfio-
  pci,host=:83:00.7,id=hostpci1.7,bus=pci.0,addr=0x11.7: vfio
  :83:00.7: hardware reports invalid configuration, MSIX PBA outside
  of specified BAR

  I discovered this bug on a Proxmox system, and I was working with a
  downstream Proxmox developer to try to fix this issue. They provided
  me with the following change to make from line 1484 of hw/vfio/pci.c:

  static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp)
    * is 0x1000, so we hard code that here.
    */
   if (vdev->vendor_id == PCI_VENDOR_ID_CHELSIO &&
  -(vdev->device_id & 0xff00) == 0x5800) {
  +((vdev->device_id & 0xff00) == 0x5800 ||
  + (vdev->device_id & 0xff00) == 0x1425)) {
   msix->pba_offset = 0x1000;
   } else if (vdev->msix_relo == OFF_AUTOPCIBAR_OFF) {
   error_setg(errp, "hardware reports invalid configuration, "

  However, I found that this did not fix the issue, so the bug appears
  to work differently than the one that was present on the T5 NICs which
  has already been patched. I have attached the output of my lspci
  -nnkvv

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1894869/+subscriptions

Re: [PATCH v9 07/12] migration/dirtyrate: Compare page hash results for recorded sampled page

2020-09-15 Thread Zheng Chuan




On 2020/9/16 0:30, Li Qiang wrote:
> Chuan Zheng  于2020年9月15日周二 上午10:34写道：
>>
>> Compare page hash results for recorded sampled page.
>>
>> Signed-off-by: Chuan Zheng 
>> Signed-off-by: YanYing Zhuang 
>> Reviewed-by: Dr. David Alan Gilbert 
>> ---
>>  migration/dirtyrate.c | 63 
>> +++
>>  1 file changed, 63 insertions(+)
>>
>> diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c
>> index 5e6eedf..2d48eb8 100644
>> --- a/migration/dirtyrate.c
>> +++ b/migration/dirtyrate.c
>> @@ -177,6 +177,69 @@ out:
>>  return ret;
>>  }
>>
>> +static void calc_page_dirty_rate(struct RamblockDirtyInfo *info)
>> +{
>> +uint32_t crc;
>> +int i;
>> +
>> +for (i = 0; i < info->sample_pages_count; i++) {
>> +crc = get_ramblock_vfn_hash(info, info->sample_page_vfn[i]);
>> +if (crc != info->hash_result[i]) {
>> +info->sample_dirty_count++;
>> +}
>> +}
>> +}
>> +
>> +static struct RamblockDirtyInfo *
>> +find_page_matched(RAMBlock *block, int count,
>> +  struct RamblockDirtyInfo *infos)
>> +{
>> +int i;
>> +struct RamblockDirtyInfo *matched;
>> +
>> +for (i = 0; i <= count; i++) {
> 
> 'i < count'?
> 
Oops, it should be.
Will fix it asap.

>> +if (!strcmp(infos[i].idstr, qemu_ram_get_idstr(block))) {
>> +break;
>> +}
>> +}
>> +
>> +if (i == count) {
>> +return NULL;
>> +}
>> +
>> +if (infos[i].ramblock_addr != qemu_ram_get_host_addr(block) ||
>> +infos[i].ramblock_pages !=
>> +(qemu_ram_get_used_length(block) >> TARGET_PAGE_BITS)) {
>> +return NULL;
>> +}
>> +
>> +matched = [i];
>> +
>> +return matched;
>> +}
>> +
>> +static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
>> +  int block_count)
>> +{
>> +struct RamblockDirtyInfo *block_dinfo = NULL;
>> +RAMBlock *block = NULL;
>> +
>> +RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> +block_dinfo = find_page_matched(block, block_count, info);
>> +if (block_dinfo == NULL) {
>> +continue;
>> +}
>> +calc_page_dirty_rate(block_dinfo);
>> +update_dirtyrate_stat(block_dinfo);
>> +}
>> +
>> +if (DirtyStat.total_sample_count == 0) {
>> +return false;
>> +}
>> +
>> +return true;
>> +}
>> +
>>  static void calculate_dirtyrate(struct DirtyRateConfig config)
>>  {
>>  /* todo */
>> --
>> 1.8.3.1
>>
> .
>

Re: [PATCH v2 1/6] util/oslib-win32: Use _aligned_malloc for qemu_try_memalign

2020-09-15 Thread Stefan Weil

Am 16.09.20 um 02:46 schrieb Richard Henderson:

> We do not need or want to be allocating page sized quanta.
>
> Signed-off-by: Richard Henderson 
> ---
> Cc: Stefan Weil 
> ---
>  util/oslib-win32.c | 10 +++---
>  1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/util/oslib-win32.c b/util/oslib-win32.c
> index c654dafd93..8d838bf342 100644
> --- a/util/oslib-win32.c
> +++ b/util/oslib-win32.c
> @@ -56,10 +56,8 @@ void *qemu_try_memalign(size_t alignment, size_t size)
>  {
>  void *ptr;
>  
> -if (!size) {
> -abort();
> -}
> -ptr = VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_READWRITE);
> +assert(size != 0);
> +ptr = _aligned_malloc(alignment, size);
>  trace_qemu_memalign(alignment, size, ptr);
>  return ptr;
>  }
> @@ -93,9 +91,7 @@ void *qemu_anon_ram_alloc(size_t size, uint64_t *align, 
> bool shared)
>  void qemu_vfree(void *ptr)
>  {
>  trace_qemu_vfree(ptr);
> -if (ptr) {
> -VirtualFree(ptr, 0, MEM_RELEASE);
> -}
> +_aligned_free(ptr);
>  }
>  
>  void qemu_anon_ram_free(void *ptr, size_t size)


According to the documentation, malloc.h should be included for
_aligned_malloc. See
https://docs.microsoft.com/de-de/cpp/c-runtime-library/reference/aligned-malloc?view=vs-2019

I'd also use g_assert instead of assert to make sure that it is not
removed by NDEBUG.

Thanks,

Stefan

Re: [PATCH 0/5] qom: Allow object to be aligned

2020-09-15 Thread Stefan Weil

Am 16.09.20 um 00:47 schrieb Richard Henderson:

> On 9/15/20 10:46 AM, Richard Henderson wrote:
>> We already have a function that can alloc with alignment,
>> but we need to pass this down from the structure.  We also
>> don't want to use this function unconditionally, because
>> the windows version does page allocation, which would be
>> overkill for the vast majority of the objects allocated.
> Stefan, why are we using VirtualAlloc in util/oslib-win32.c, 
> qemu_try_memalign,
> instead of _aligned_malloc?

I think we use it because Fabrice introduced VirtualAlloc in commit
6e4255f6a65091fbe7d17bfda546e2aa1b72f9a6 (with a comment "FIXME: this is
not exactly optimal solution [...]"). Maybe there was no _aligned_malloc
available 15 years ago. Commit 33f002714be2ed58ed05ae3870d5ea6915df4b47
then reused VirtualAlloc for qemu_try_memalign (without a FIXME comment).

Obviously nobody cared to find a better solution until now.

Stefan

[Bug 1883984] Update Released

2020-09-15 Thread Chris Halse Rogers

The verification of the Stable Release Update for qemu has completed
successfully and the package is now being released to -updates.
Subsequently, the Ubuntu Stable Release Updates Team is being
unsubscribed and will not receive messages about this bug report.  In
the event that you encounter a regression using the package from
-updates please report a new bug using ubuntu-bug and tag the bug report
regression-update so we can easily find any regressions.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1883984

Title:
  QEMU S/390x sqxbr (128-bit IEEE 754 square root) crashes qemu-system-
  s390x

Status in QEMU:
  Fix Released
Status in qemu package in Ubuntu:
  Fix Released
Status in qemu source package in Focal:
  Fix Released

Bug description:
  [Impact]

   * An instruction was described wrong so that on usage the program would 
 crash.

  [Test Case]

   * Run s390x in emulation and there use this program:
 For simplicity and speed you can use KVM guest as usual on s390x, that 
 after prep of the test you run in qemu-tcg like:

 $ sudo qemu-system-s390x -machine s390-ccw-virtio,accel=tcg -cpu 
max,zpci=on -serial mon:stdio -display none -m 4096 -nic 
user,model=virtio,hostfwd=tcp::-:22 -drive 
file=/var/lib/uvtool/libvirt/images/focal-sqxbr.qcow,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
 -device 
virtio-blk-ccw,devno=fe.0.0001,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,scsi=off
 Obviously is you have no s390x access you need to use emulation right 
 away.

   * Build and run failing program
 $ sudo apt install clang
 $ cat > bug-sqrtl-one-line.c << EOF
  int main(void) { volatile long double x, r; x = 4.0L; __asm__ 
  __volatile__("sqxbr %0, %1" : "=f" (r) : "f" (x)); return (0);}
  EOF
 $ cc bug-sqrtl-one-line.c
 $ ./a.out
 Segmentation fault (core dumped)

 qemu is dead by now as long as the bug is present

  [Regression Potential]

   * The change only modifies 128 bit square root on s390x so regressions
 should be limited to exactly that - which formerly before this fix was 
 a broken instruction.

  [Other Info]
   
   * n/a

  ---

  In porting software to guest Ubuntu 18.04 and 20.04 VMs for S/390x, I 
discovered
  that some of my own numerical programs, and also a GNU configure script for at
  least one package with CC=clang, would cause an instant crash of the VM, 
sometimes
  also destroying recently opened files, and producing long strings of NUL 
characters
  in /var/log/syslog in the S/390 guest O/S.

  Further detective work narrowed the cause of the crash down to a single IBM 
S/390
  instruction: sqxbr (128-bit IEEE 754 square root).  Here is a one-line program
  that when compiled and run on a VM hosted on QEMUcc emulator version 4.2.0
  (Debian 1:4.2-3ubuntu6.1) [hosted on Ubuntu 20.04 on a Dell Precision 7920
  workstation with an Intel Xeon Platinum 8253 CPU],  and also on QEMU emulator
  version 5.0.0, reproducibly produces a VM crash under qemu-system-s390x.

  % cat bug-sqrtl-one-line.c
  int main(void) { volatile long double x, r; x = 4.0L; __asm__ 
__volatile__("sqxbr %0, %1" : "=f" (r) : "f" (x)); return (0);}

  % cc bug-sqrtl-one-line.c && ./a.out
  Segmentation fault (core dumped)

  The problem code may be the function float128_sqrt() defined in 
qemu-5.0.0/fpu/softfloat.c
  starting at line 7619.  I have NOT attempted to run the qemu-system-s390x 
executable
  under a debugger.  However, I observe that S/390 is the only CPU family that 
I know of,
  except possibly for a Fujitsu SPARC-64, that has a 128-bit square root in 
hardware.
  Thus, this instruction bug may not have been seen before.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1883984/+subscriptions

[Bug 1883984] Re: QEMU S/390x sqxbr (128-bit IEEE 754 square root) crashes qemu-system-s390x

2020-09-15 Thread Launchpad Bug Tracker

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.5

---
qemu (1:4.2-3ubuntu6.5) focal; urgency=medium

  * further stabilize qemu by importing patches of qemu v4.2.1
Fixes (LP: #1891203) and (LP: #1891877)
- d/p/stable/lp-1891877-*
- as part of the stabilization this also fixes an
  riscv emulation issue due to the CVE-2020-13754 fixes via
  d/p/ubuntu/hw-riscv-Allow-64-bit-access-to-SiFive-CLINT.patch
  * fix s390x SQXBR emulation (LP: #1883984)
- d/p/ubuntu/lp-1883984-target-s390x-Fix-SQXBR.patch
  * fix -no-reboot for s390x protvirt guests (LP: #1890154)
- d/p/ubuntu/lp-1890154-s390x-protvirt-allow-to-IPL-secure-guests-with-*

 -- Christian Ehrhardt   Wed, 19 Aug
2020 13:40:49 +0200

** Changed in: qemu (Ubuntu Focal)
   Status: Fix Committed => Fix Released

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-13754

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1883984

Title:
  QEMU S/390x sqxbr (128-bit IEEE 754 square root) crashes qemu-system-
  s390x

Status in QEMU:
  Fix Released
Status in qemu package in Ubuntu:
  Fix Released
Status in qemu source package in Focal:
  Fix Released

Bug description:
  [Impact]

   * An instruction was described wrong so that on usage the program would 
 crash.

  [Test Case]

   * Run s390x in emulation and there use this program:
 For simplicity and speed you can use KVM guest as usual on s390x, that 
 after prep of the test you run in qemu-tcg like:

 $ sudo qemu-system-s390x -machine s390-ccw-virtio,accel=tcg -cpu 
max,zpci=on -serial mon:stdio -display none -m 4096 -nic 
user,model=virtio,hostfwd=tcp::-:22 -drive 
file=/var/lib/uvtool/libvirt/images/focal-sqxbr.qcow,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
 -device 
virtio-blk-ccw,devno=fe.0.0001,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,scsi=off
 Obviously is you have no s390x access you need to use emulation right 
 away.

   * Build and run failing program
 $ sudo apt install clang
 $ cat > bug-sqrtl-one-line.c << EOF
  int main(void) { volatile long double x, r; x = 4.0L; __asm__ 
  __volatile__("sqxbr %0, %1" : "=f" (r) : "f" (x)); return (0);}
  EOF
 $ cc bug-sqrtl-one-line.c
 $ ./a.out
 Segmentation fault (core dumped)

 qemu is dead by now as long as the bug is present

  [Regression Potential]

   * The change only modifies 128 bit square root on s390x so regressions
 should be limited to exactly that - which formerly before this fix was 
 a broken instruction.

  [Other Info]
   
   * n/a

  ---

  In porting software to guest Ubuntu 18.04 and 20.04 VMs for S/390x, I 
discovered
  that some of my own numerical programs, and also a GNU configure script for at
  least one package with CC=clang, would cause an instant crash of the VM, 
sometimes
  also destroying recently opened files, and producing long strings of NUL 
characters
  in /var/log/syslog in the S/390 guest O/S.

  Further detective work narrowed the cause of the crash down to a single IBM 
S/390
  instruction: sqxbr (128-bit IEEE 754 square root).  Here is a one-line program
  that when compiled and run on a VM hosted on QEMUcc emulator version 4.2.0
  (Debian 1:4.2-3ubuntu6.1) [hosted on Ubuntu 20.04 on a Dell Precision 7920
  workstation with an Intel Xeon Platinum 8253 CPU],  and also on QEMU emulator
  version 5.0.0, reproducibly produces a VM crash under qemu-system-s390x.

  % cat bug-sqrtl-one-line.c
  int main(void) { volatile long double x, r; x = 4.0L; __asm__ 
__volatile__("sqxbr %0, %1" : "=f" (r) : "f" (x)); return (0);}

  % cc bug-sqrtl-one-line.c && ./a.out
  Segmentation fault (core dumped)

  The problem code may be the function float128_sqrt() defined in 
qemu-5.0.0/fpu/softfloat.c
  starting at line 7619.  I have NOT attempted to run the qemu-system-s390x 
executable
  under a debugger.  However, I observe that S/390 is the only CPU family that 
I know of,
  except possibly for a Fujitsu SPARC-64, that has a 128-bit square root in 
hardware.
  Thus, this instruction bug may not have been seen before.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1883984/+subscriptions

[PATCH V9 6/6] hw/mips: Add Loongson-3 machine support

2020-09-15 Thread Huacai Chen

Add Loongson-3 based machine support, it use liointc as the interrupt
controler and use GPEX as the pci controller. Currently it can work with
both TCG and KVM.

As the machine model is not based on any exiting physical hardware, the
name of the machine is "loongson3-virt". It may be superseded in future
by a real machine model. If this happens, then a regular deprecation
procedure shall occur for "loongson3-virt" machine.

We now already have a full functional Linux kernel (based on Linux-5.4.x
LTS, the kvm host side and guest side have both been upstream for Linux-
5.9, but Linux-5.9 has not been released yet) here:

https://github.com/chenhuacai/linux

Of course the upstream kernel is also usable (though it is "unstable"
now):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

How to use QEMU/Loongson-3?
1, Download kernel source from the above URL;
2, Build a kernel with arch/mips/configs/loongson3_defconfig;
3, Boot a Loongson-3A4000 host with this kernel (for KVM mode);
4, Build QEMU-master with this patchset;
5, modprobe kvm (only necessary for KVM mode);
6, Use QEMU with TCG:
   qemu-system-mips64el -M loongson3-virt,accel=tcg -cpu Loongson-3A1000 
-kernel  -append ...
   Use QEMU with KVM:
   qemu-system-mips64el -M loongson3-virt,accel=kvm -cpu Loongson-3A4000 
-kernel  -append ...

   The "-cpu" parameter is optional here and QEMU will use the correct type for 
TCG/KVM automatically.

Signed-off-by: Huacai Chen 
Co-developed-by: Jiaxun Yang 
---
 default-configs/mips64el-softmmu.mak |   1 +
 hw/mips/Kconfig  |  11 +
 hw/mips/fw_cfg.c |  35 ++
 hw/mips/fw_cfg.h |  19 +
 hw/mips/loongson3_virt.c | 956 +++
 hw/mips/meson.build  |   3 +-
 6 files changed, 1024 insertions(+), 1 deletion(-)
 create mode 100644 hw/mips/fw_cfg.c
 create mode 100644 hw/mips/fw_cfg.h
 create mode 100644 hw/mips/loongson3_virt.c

diff --git a/default-configs/mips64el-softmmu.mak 
b/default-configs/mips64el-softmmu.mak
index 9f8a3ef..26c660a 100644
--- a/default-configs/mips64el-softmmu.mak
+++ b/default-configs/mips64el-softmmu.mak
@@ -3,6 +3,7 @@
 include mips-softmmu-common.mak
 CONFIG_IDE_VIA=y
 CONFIG_FULOONG=y
+CONFIG_LOONGSON3V=y
 CONFIG_ATI_VGA=y
 CONFIG_RTL8139_PCI=y
 CONFIG_JAZZ=y
diff --git a/hw/mips/Kconfig b/hw/mips/Kconfig
index 67d39c5..cc5609b 100644
--- a/hw/mips/Kconfig
+++ b/hw/mips/Kconfig
@@ -45,6 +45,17 @@ config FULOONG
 bool
 select PCI_BONITO
 
+config LOONGSON3V
+bool
+select PCKBD
+select SERIAL
+select GOLDFISH_RTC
+select LOONGSON_LIOINTC
+select PCI_EXPRESS_GENERIC_BRIDGE
+select VIRTIO_VGA
+select QXL if SPICE
+select MSI_NONBROKEN
+
 config MIPS_CPS
 bool
 select PTIMER
diff --git a/hw/mips/fw_cfg.c b/hw/mips/fw_cfg.c
new file mode 100644
index 000..67c4a74
--- /dev/null
+++ b/hw/mips/fw_cfg.c
@@ -0,0 +1,35 @@
+/*
+ * QEMU fw_cfg helpers (MIPS specific)
+ *
+ * Copyright (c) 2020 Lemote, Inc.
+ *
+ * Author:
+ *   Huacai Chen (che...@lemote.com)
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/mips/fw_cfg.h"
+#include "hw/nvram/fw_cfg.h"
+
+const char *fw_cfg_arch_key_name(uint16_t key)
+{
+static const struct {
+uint16_t key;
+const char *name;
+} fw_cfg_arch_wellknown_keys[] = {
+{FW_CFG_MACHINE_VERSION, "machine_version"},
+{FW_CFG_CPU_FREQ, "cpu_frequency"},
+};
+
+for (size_t i = 0; i < ARRAY_SIZE(fw_cfg_arch_wellknown_keys); i++) {
+if (fw_cfg_arch_wellknown_keys[i].key == key) {
+return fw_cfg_arch_wellknown_keys[i].name;
+}
+}
+return NULL;
+}
diff --git a/hw/mips/fw_cfg.h b/hw/mips/fw_cfg.h
new file mode 100644
index 000..e317d5b
--- /dev/null
+++ b/hw/mips/fw_cfg.h
@@ -0,0 +1,19 @@
+/*
+ * QEMU fw_cfg helpers (MIPS specific)
+ *
+ * Copyright (c) 2020 Huacai Chen
+ *
+ * SPDX-License-Identifier: MIT
+ */
+
+#ifndef HW_MIPS_FW_CFG_H
+#define HW_MIPS_FW_CFG_H
+
+#include "hw/boards.h"
+#include "hw/nvram/fw_cfg.h"
+
+/* Data for BIOS to identify machine */
+#define FW_CFG_MACHINE_VERSION  (FW_CFG_ARCH_LOCAL + 0)
+#define FW_CFG_CPU_FREQ (FW_CFG_ARCH_LOCAL + 1)
+
+#endif
diff --git a/hw/mips/loongson3_virt.c b/hw/mips/loongson3_virt.c
new file mode 100644
index 000..14d3024
--- /dev/null
+++ b/hw/mips/loongson3_virt.c
@@ -0,0 +1,956 @@
+/*
+ * Generic Loongson-3 Platform support
+ *
+ * Copyright (c) 2017-2020 Huacai Chen (che...@lemote.com)
+ * Copyright (c) 2017-2020 Jiaxun Yang 
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your

[PATCH V9 5/6] target/mips: Add loongson-ext lsdc2 group of instructions

2020-09-15 Thread Huacai Chen

From: Jiaxun Yang 

LDC2/SDC2 opcodes have been rewritten as "load & store with offset"
group of instructions by loongson-ext ASE.

This patch add implementation of these instructions:
gslbx: load 1 bytes to GPR
gslhx: load 2 bytes to GPR
gslwx: load 4 bytes to GPR
gsldx: load 8 bytes to GPR
gslwxc1: load 4 bytes to FPR
gsldxc1: load 8 bytes to FPR
gssbx: store 1 bytes from GPR
gsshx: store 2 bytes from GPR
gsswx: store 4 bytes from GPR
gssdx: store 8 bytes from GPR
gsswxc1: store 4 bytes from FPR
gssdxc1: store 8 bytes from FPR

Details of Loongson-EXT is here:
https://github.com/FlyGoat/loongson-insn/blob/master/loongson-ext.md

Signed-off-by: Huacai Chen 
Signed-off-by: Jiaxun Yang 
---
 target/mips/translate.c | 179 
 1 file changed, 179 insertions(+)

diff --git a/target/mips/translate.c b/target/mips/translate.c
index 2bcf1ce..c7ec8cf 100644
--- a/target/mips/translate.c
+++ b/target/mips/translate.c
@@ -484,6 +484,24 @@ enum {
 OPC_GSSDRC1 = 0x7 | OPC_GSSHFS,
 };
 
+/* Loongson EXT LDC2/SDC2 opcodes */
+#define MASK_LOONGSON_LSDC2(op)   (MASK_OP_MAJOR(op) | (op & 0x7))
+
+enum {
+OPC_GSLBX  = 0x0 | OPC_LDC2,
+OPC_GSLHX  = 0x1 | OPC_LDC2,
+OPC_GSLWX  = 0x2 | OPC_LDC2,
+OPC_GSLDX  = 0x3 | OPC_LDC2,
+OPC_GSLWXC1= 0x6 | OPC_LDC2,
+OPC_GSLDXC1= 0x7 | OPC_LDC2,
+OPC_GSSBX  = 0x0 | OPC_SDC2,
+OPC_GSSHX  = 0x1 | OPC_SDC2,
+OPC_GSSWX  = 0x2 | OPC_SDC2,
+OPC_GSSDX  = 0x3 | OPC_SDC2,
+OPC_GSSWXC1= 0x6 | OPC_SDC2,
+OPC_GSSDXC1= 0x7 | OPC_SDC2,
+};
+
 /* BSHFL opcodes */
 #define MASK_BSHFL(op)  (MASK_SPECIAL3(op) | (op & (0x1F << 6)))
 
@@ -6166,6 +6184,165 @@ static void gen_loongson_lswc2(DisasContext *ctx, int 
rt,
 tcg_temp_free(t0);
 }
 
+/* Loongson EXT LDC2/SDC2 */
+static void gen_loongson_lsdc2(DisasContext *ctx, int rt,
+int rs, int rd)
+{
+int offset = (int8_t)(ctx->opcode >> 3);
+uint32_t opc = MASK_LOONGSON_LSDC2(ctx->opcode);
+TCGv t0, t1;
+TCGv_i32 fp0;
+
+/* Pre-conditions */
+switch (opc) {
+case OPC_GSLBX:
+case OPC_GSLHX:
+case OPC_GSLWX:
+case OPC_GSLDX:
+/* prefetch, implement as NOP */
+if (rt == 0) {
+return;
+}
+break;
+case OPC_GSSBX:
+case OPC_GSSHX:
+case OPC_GSSWX:
+case OPC_GSSDX:
+break;
+case OPC_GSLWXC1:
+#if defined(TARGET_MIPS64)
+case OPC_GSLDXC1:
+#endif
+check_cp1_enabled(ctx);
+/* prefetch, implement as NOP */
+if (rt == 0) {
+return;
+}
+break;
+case OPC_GSSWXC1:
+#if defined(TARGET_MIPS64)
+case OPC_GSSDXC1:
+#endif
+check_cp1_enabled(ctx);
+break;
+default:
+MIPS_INVAL("loongson_lsdc2");
+generate_exception_end(ctx, EXCP_RI);
+return;
+break;
+}
+
+t0 = tcg_temp_new();
+
+gen_base_offset_addr(ctx, t0, rs, offset);
+gen_op_addr_add(ctx, t0, cpu_gpr[rd], t0);
+
+switch (opc) {
+case OPC_GSLBX:
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_SB);
+gen_store_gpr(t0, rt);
+break;
+case OPC_GSLHX:
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TESW |
+ctx->default_tcg_memop_mask);
+gen_store_gpr(t0, rt);
+break;
+case OPC_GSLWX:
+gen_base_offset_addr(ctx, t0, rs, offset);
+if (rd) {
+gen_op_addr_add(ctx, t0, cpu_gpr[rd], t0);
+}
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TESL |
+ctx->default_tcg_memop_mask);
+gen_store_gpr(t0, rt);
+break;
+#if defined(TARGET_MIPS64)
+case OPC_GSLDX:
+gen_base_offset_addr(ctx, t0, rs, offset);
+if (rd) {
+gen_op_addr_add(ctx, t0, cpu_gpr[rd], t0);
+}
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_gpr(t0, rt);
+break;
+#endif
+case OPC_GSLWXC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, offset);
+if (rd) {
+gen_op_addr_add(ctx, t0, cpu_gpr[rd], t0);
+}
+fp0 = tcg_temp_new_i32();
+tcg_gen_qemu_ld_i32(fp0, t0, ctx->mem_idx, MO_TESL |
+ctx->default_tcg_memop_mask);
+gen_store_fpr32(ctx, fp0, rt);
+tcg_temp_free_i32(fp0);
+break;
+#if defined(TARGET_MIPS64)
+case OPC_GSLDXC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, offset);
+if (rd) {
+gen_op_addr_add(ctx, t0, cpu_gpr[rd], t0);
+}
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_fpr64(ctx, t0, rt);
+break;
+#endif
+case OPC_GSSBX:
+t1 =

[PATCH V9 4/6] target/mips: Add loongson-ext lswc2 group of instructions (Part 2)

2020-09-15 Thread Huacai Chen

From: Jiaxun Yang 

LWC2 & SWC2 have been rewritten by Loongson EXT vendor ASE
as "load/store quad word" and "shifted load/store" groups of
instructions.

This patch add implementation of these instructions:
gslwlc1: similar to lwl but RT is FPR instead of GPR
gslwrc1: similar to lwr but RT is FPR instead of GPR
gsldlc1: similar to ldl but RT is FPR instead of GPR
gsldrc1: similar to ldr but RT is FPR instead of GPR
gsswlc1: similar to swl but RT is FPR instead of GPR
gsswrc1: similar to swr but RT is FPR instead of GPR
gssdlc1: similar to sdl but RT is FPR instead of GPR
gssdrc1: similar to sdr but RT is FPR instead of GPR

Details of Loongson-EXT is here:
https://github.com/FlyGoat/loongson-insn/blob/master/loongson-ext.md

Signed-off-by: Huacai Chen 
Signed-off-by: Jiaxun Yang 
---
 target/mips/translate.c | 177 
 1 file changed, 177 insertions(+)

diff --git a/target/mips/translate.c b/target/mips/translate.c
index 08d51e1..2bcf1ce 100644
--- a/target/mips/translate.c
+++ b/target/mips/translate.c
@@ -471,6 +471,19 @@ enum {
 OPC_GSSHFS  = OPC_SWC2,
 };
 
+/* Loongson EXT shifted load/store opcodes */
+#define MASK_LOONGSON_GSSHFLS(op) (MASK_OP_MAJOR(op) | (op & 0xc03f))
+enum {
+OPC_GSLWLC1 = 0x4 | OPC_GSSHFL,
+OPC_GSLWRC1 = 0x5 | OPC_GSSHFL,
+OPC_GSLDLC1 = 0x6 | OPC_GSSHFL,
+OPC_GSLDRC1 = 0x7 | OPC_GSSHFL,
+OPC_GSSWLC1 = 0x4 | OPC_GSSHFS,
+OPC_GSSWRC1 = 0x5 | OPC_GSSHFS,
+OPC_GSSDLC1 = 0x6 | OPC_GSSHFS,
+OPC_GSSDRC1 = 0x7 | OPC_GSSHFS,
+};
+
 /* BSHFL opcodes */
 #define MASK_BSHFL(op)  (MASK_SPECIAL3(op) | (op & (0x1F << 6)))
 
@@ -5981,6 +5994,170 @@ static void gen_loongson_lswc2(DisasContext *ctx, int 
rt,
 tcg_temp_free(t1);
 break;
 #endif
+case OPC_GSSHFL:
+switch (MASK_LOONGSON_GSSHFLS(ctx->opcode)) {
+case OPC_GSLWLC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, shf_offset);
+t1 = tcg_temp_new();
+tcg_gen_qemu_ld_tl(t1, t0, ctx->mem_idx, MO_UB);
+tcg_gen_andi_tl(t1, t0, 3);
+#ifndef TARGET_WORDS_BIGENDIAN
+tcg_gen_xori_tl(t1, t1, 3);
+#endif
+tcg_gen_shli_tl(t1, t1, 3);
+tcg_gen_andi_tl(t0, t0, ~3);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEUL);
+tcg_gen_shl_tl(t0, t0, t1);
+t2 = tcg_const_tl(-1);
+tcg_gen_shl_tl(t2, t2, t1);
+fp0 = tcg_temp_new_i32();
+gen_load_fpr32(ctx, fp0, rt);
+tcg_gen_ext_i32_tl(t1, fp0);
+tcg_gen_andc_tl(t1, t1, t2);
+tcg_temp_free(t2);
+tcg_gen_or_tl(t0, t0, t1);
+tcg_temp_free(t1);
+#if defined(TARGET_MIPS64)
+tcg_gen_extrl_i64_i32(fp0, t0);
+#else
+tcg_gen_ext32s_tl(fp0, t0);
+#endif
+gen_store_fpr32(ctx, fp0, rt);
+tcg_temp_free_i32(fp0);
+break;
+case OPC_GSLWRC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, shf_offset);
+t1 = tcg_temp_new();
+tcg_gen_qemu_ld_tl(t1, t0, ctx->mem_idx, MO_UB);
+tcg_gen_andi_tl(t1, t0, 3);
+#ifdef TARGET_WORDS_BIGENDIAN
+tcg_gen_xori_tl(t1, t1, 3);
+#endif
+tcg_gen_shli_tl(t1, t1, 3);
+tcg_gen_andi_tl(t0, t0, ~3);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEUL);
+tcg_gen_shr_tl(t0, t0, t1);
+tcg_gen_xori_tl(t1, t1, 31);
+t2 = tcg_const_tl(0xfffeull);
+tcg_gen_shl_tl(t2, t2, t1);
+fp0 = tcg_temp_new_i32();
+gen_load_fpr32(ctx, fp0, rt);
+tcg_gen_ext_i32_tl(t1, fp0);
+tcg_gen_and_tl(t1, t1, t2);
+tcg_temp_free(t2);
+tcg_gen_or_tl(t0, t0, t1);
+tcg_temp_free(t1);
+#if defined(TARGET_MIPS64)
+tcg_gen_extrl_i64_i32(fp0, t0);
+#else
+tcg_gen_ext32s_tl(fp0, t0);
+#endif
+gen_store_fpr32(ctx, fp0, rt);
+tcg_temp_free_i32(fp0);
+break;
+#if defined(TARGET_MIPS64)
+case OPC_GSLDLC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, shf_offset);
+t1 = tcg_temp_new();
+tcg_gen_qemu_ld_tl(t1, t0, ctx->mem_idx, MO_UB);
+tcg_gen_andi_tl(t1, t0, 7);
+#ifndef TARGET_WORDS_BIGENDIAN
+tcg_gen_xori_tl(t1, t1, 7);
+#endif
+tcg_gen_shli_tl(t1, t1, 3);
+tcg_gen_andi_tl(t0, t0, ~7);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ);
+tcg_gen_shl_tl(t0, t0, t1);
+t2 = tcg_const_tl(-1);
+tcg_gen_shl_tl(t2, t2, t1);
+gen_load_fpr64(ctx, t1, rt);
+tcg_gen_andc_tl(t1, t1, t2);
+tcg_temp_free(t2);
+tcg_gen_or_tl(t0, t0, t1);
+

[PATCH V9 2/6] target/mips: Fix PageMask with variable page size

2020-09-15 Thread Huacai Chen

From: Jiaxun Yang 

Our current code assumed the target page size is always 4k
when handling PageMask and VPN2, however, variable page size
was just added to mips target and that's nolonger true.

Signed-off-by: Huacai Chen 
Signed-off-by: Jiaxun Yang 
---
 target/mips/cp0_helper.c | 36 +---
 target/mips/cpu.h|  1 +
 2 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/target/mips/cp0_helper.c b/target/mips/cp0_helper.c
index de64add..62bcb97 100644
--- a/target/mips/cp0_helper.c
+++ b/target/mips/cp0_helper.c
@@ -867,13 +867,35 @@ void helper_mtc0_memorymapid(CPUMIPSState *env, 
target_ulong arg1)
 
 void update_pagemask(CPUMIPSState *env, target_ulong arg1, int32_t *pagemask)
 {
-uint64_t mask = arg1 >> (TARGET_PAGE_BITS + 1);
-if (!(env->insn_flags & ISA_MIPS32R6) || (arg1 == ~0) ||
-(mask == 0x || mask == 0x0003 || mask == 0x000F ||
- mask == 0x003F || mask == 0x00FF || mask == 0x03FF ||
- mask == 0x0FFF || mask == 0x3FFF || mask == 0x)) {
-env->CP0_PageMask = arg1 & (0x1FFF & (TARGET_PAGE_MASK << 1));
+unsigned long mask;
+int maskbits;
+
+if (env->insn_flags & ISA_MIPS32R6) {
+return;
+}
+/* Don't care MASKX as we don't support 1KB page */
+mask = extract32((uint32_t)arg1, CP0PM_MASK, 16);
+maskbits = find_first_zero_bit(, 32);
+
+/* Ensure no more set bit after first zero */
+if (mask >> maskbits) {
+goto invalid;
+}
+/* We don't support VTLB entry smaller than target page */
+if ((maskbits + 12) < TARGET_PAGE_BITS) {
+goto invalid;
 }
+env->CP0_PageMask = mask << CP0PM_MASK;
+
+return;
+
+invalid:
+/*
+ * When invalid, ensure the value is bigger tan or equel to
+ * the minimal but smaller than or equel to the maxium.
+ */
+maskbits = MIN(16, MAX(maskbits, TARGET_PAGE_BITS - 12));
+env->CP0_PageMask = ((1 << (16 + 1)) - 1) << CP0PM_MASK;
 }
 
 void helper_mtc0_pagemask(CPUMIPSState *env, target_ulong arg1)
@@ -1104,7 +1126,7 @@ void helper_mthc0_saar(CPUMIPSState *env, target_ulong 
arg1)
 void helper_mtc0_entryhi(CPUMIPSState *env, target_ulong arg1)
 {
 target_ulong old, val, mask;
-mask = (TARGET_PAGE_MASK << 1) | env->CP0_EntryHi_ASID_mask;
+mask = ~((1 << 14) - 1) | env->CP0_EntryHi_ASID_mask;
 if (((env->CP0_Config4 >> CP0C4_IE) & 0x3) >= 2) {
 mask |= 1 << CP0EnHi_EHINV;
 }
diff --git a/target/mips/cpu.h b/target/mips/cpu.h
index 7cf7f52..9c8bb23 100644
--- a/target/mips/cpu.h
+++ b/target/mips/cpu.h
@@ -618,6 +618,7 @@ struct CPUMIPSState {
  * CP0 Register 5
  */
 int32_t CP0_PageMask;
+#define CP0PM_MASK 13
 int32_t CP0_PageGrain_rw_bitmask;
 int32_t CP0_PageGrain;
 #define CP0PG_RIE 31
-- 
2.7.0

[PATCH V9 3/6] target/mips: Add loongson-ext lswc2 group of instructions (Part 1)

2020-09-15 Thread Huacai Chen

From: Jiaxun Yang 

LWC2 & SWC2 have been rewritten by Loongson EXT vendor ASE
as "load/store quad word" and "shifted load/store" groups of
instructions.

This patch add implementation of these instructions:
gslq: load 16 bytes to GPR
gssq: store 16 bytes from GPR
gslqc1: load 16 bytes to FPR
gssqc1: store 16 bytes from FPR

Details of Loongson-EXT is here:
https://github.com/FlyGoat/loongson-insn/blob/master/loongson-ext.md

Signed-off-by: Huacai Chen 
Signed-off-by: Jiaxun Yang 
---
 target/mips/translate.c | 81 +
 1 file changed, 81 insertions(+)

diff --git a/target/mips/translate.c b/target/mips/translate.c
index 398edf7..08d51e1 100644
--- a/target/mips/translate.c
+++ b/target/mips/translate.c
@@ -460,6 +460,17 @@ enum {
 R6_OPC_SCD = 0x27 | OPC_SPECIAL3,
 };
 
+/* Loongson EXT load/store quad word opcodes */
+#define MASK_LOONGSON_GSLSQ(op)   (MASK_OP_MAJOR(op) | (op & 0x8020))
+enum {
+OPC_GSLQ= 0x0020 | OPC_LWC2,
+OPC_GSLQC1  = 0x8020 | OPC_LWC2,
+OPC_GSSHFL  = OPC_LWC2,
+OPC_GSSQ= 0x0020 | OPC_SWC2,
+OPC_GSSQC1  = 0x8020 | OPC_SWC2,
+OPC_GSSHFS  = OPC_SWC2,
+};
+
 /* BSHFL opcodes */
 #define MASK_BSHFL(op)  (MASK_SPECIAL3(op) | (op & (0x1F << 6)))
 
@@ -5910,6 +5921,74 @@ no_rd:
 tcg_temp_free_i64(t1);
 }
 
+static void gen_loongson_lswc2(DisasContext *ctx, int rt,
+int rs, int rd)
+{
+TCGv t0, t1, t2;
+TCGv_i32 fp0;
+int lsq_offset = ((int)((ctx->opcode >> 6) & 0x1ff) << 23) >> 19;
+int lsq_rt1 = ctx->opcode & 0x1f;
+int shf_offset = (int8_t)(ctx->opcode >> 6);
+
+t0 = tcg_temp_new();
+
+switch (MASK_LOONGSON_GSLSQ(ctx->opcode)) {
+#if defined(TARGET_MIPS64)
+case OPC_GSLQ:
+gen_base_offset_addr(ctx, t0, rs, lsq_offset);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_gpr(t0, rt);
+gen_base_offset_addr(ctx, t0, rs, lsq_offset + 8);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_gpr(t0, lsq_rt1);
+break;
+case OPC_GSLQC1:
+check_cp1_enabled(ctx);
+gen_base_offset_addr(ctx, t0, rs, lsq_offset);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_fpr64(ctx, t0, rt);
+gen_base_offset_addr(ctx, t0, rs, lsq_offset + 8);
+tcg_gen_qemu_ld_tl(t0, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_store_fpr64(ctx, t0, lsq_rt1);
+break;
+case OPC_GSSQ:
+t1 = tcg_temp_new();
+gen_base_offset_addr(ctx, t0, rs, lsq_offset);
+gen_load_gpr(t1, rt);
+tcg_gen_qemu_st_tl(t1, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_base_offset_addr(ctx, t0, rs, lsq_offset + 8);
+gen_load_gpr(t1, lsq_rt1);
+tcg_gen_qemu_st_tl(t1, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+tcg_temp_free(t1);
+break;
+case OPC_GSSQC1:
+check_cp1_enabled(ctx);
+t1 = tcg_temp_new();
+gen_base_offset_addr(ctx, t0, rs, lsq_offset);
+gen_load_fpr64(ctx, t1, rt);
+tcg_gen_qemu_st_tl(t1, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+gen_base_offset_addr(ctx, t0, rs, lsq_offset + 8);
+gen_load_fpr64(ctx, t1, lsq_rt1);
+tcg_gen_qemu_st_tl(t1, t0, ctx->mem_idx, MO_TEQ |
+ctx->default_tcg_memop_mask);
+tcg_temp_free(t1);
+break;
+#endif
+default:
+MIPS_INVAL("loongson_gslsq");
+generate_exception_end(ctx, EXCP_RI);
+break;
+}
+tcg_temp_free(t0);
+}
+
 /* Traps */
 static void gen_trap(DisasContext *ctx, uint32_t opc,
  int rs, int rt, int16_t imm)
@@ -30774,6 +30853,8 @@ static void decode_opc(CPUMIPSState *env, DisasContext 
*ctx)
 /* OPC_BC, OPC_BALC */
 gen_compute_compact_branch(ctx, op, 0, 0,
sextract32(ctx->opcode << 2, 0, 28));
+} else if (ctx->insn_flags & ASE_LEXT) {
+gen_loongson_lswc2(ctx, rt, rs, rd);
 } else {
 /* OPC_LWC2, OPC_SWC2 */
 /* COP2: Not implemented. */
-- 
2.7.0

[PATCH V9 1/6] linux-headers: Update MIPS KVM type defintition

2020-09-15 Thread Huacai Chen

Update MIPS KVM type defintition from Linux 5.9-rc6.

Signed-off-by: Huacai Chen 
---
 linux-headers/linux/kvm.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index a28c366..36a480f 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -789,9 +789,10 @@ struct kvm_ppc_resize_hpt {
 #define KVM_VM_PPC_HV 1
 #define KVM_VM_PPC_PR 2
 
-/* on MIPS, 0 forces trap & emulate, 1 forces VZ ASE */
-#define KVM_VM_MIPS_TE 0
+/* on MIPS, 0 indicates auto, 1 forces VZ ASE, 2 forces trap & emulate */
+#define KVM_VM_MIPS_AUTO   0
 #define KVM_VM_MIPS_VZ 1
+#define KVM_VM_MIPS_TE 2
 
 #define KVM_S390_SIE_PAGE_OFFSET 1
 
-- 
2.7.0

[PATCH V9 0/6] mips: Add Loongson-3 machine support

2020-09-15 Thread Huacai Chen

Loongson-3 CPU family include Loongson-3A R1/R2/R3/R4 and Loongson-3B
R1/R2. Loongson-3A R1 is the oldest and its ISA is the smallest, while
Loongson-3A R4 is the newest and its ISA is almost the superset of all
others. To reduce complexity, in QEMU we just define two CPU types:

1, "Loongson-3A1000" CPU which is corresponding to Loongson-3A R1. It is
   suitable for TCG because Loongson-3A R1 has fewest ASE.
2, "Loongson-3A4000" CPU which is corresponding to Loongson-3A R4. It is
   suitable for KVM because Loongson-3A R4 has the VZ ASE.

Loongson-3 lacks English documents. I've tried to translated them with
translate.google.com, and the machine translated documents (together
with their original Chinese versions) are available here.

Loongson-3A R1 (Loongson-3A1000)
User Manual Part 1:
http://ftp.godson.ac.cn/lemote/3A1000_p1.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A1000_processor_user_manual_P1.pdf 
(Chinese Version)
User Manual Part 2:
http://ftp.godson.ac.cn/lemote/3A1000_p2.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A1000_processor_user_manual_P2.pdf 
(Chinese Version)

Loongson-3A R2 (Loongson-3A2000)
User Manual Part 1:
http://ftp.godson.ac.cn/lemote/3A2000_p1.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A2000_user1.pdf (Chinese Version)
User Manual Part 2:
http://ftp.godson.ac.cn/lemote/3A2000_p2.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A2000_user2.pdf (Chinese Version)

Loongson-3A R3 (Loongson-3A3000)
User Manual Part 1:
http://ftp.godson.ac.cn/lemote/3A3000_p1.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A3000_3B3000usermanual1.pdf (Chinese 
Version)
User Manual Part 2:
http://ftp.godson.ac.cn/lemote/3A3000_p2.pdf
http://ftp.godson.ac.cn/lemote/Loongson3A3000_3B3000usermanual2.pdf (Chinese 
Version)

Loongson-3A R4 (Loongson-3A4000)
User Manual Part 1:
http://ftp.godson.ac.cn/lemote/3A4000_p1.pdf
http://ftp.godson.ac.cn/lemote/3A4000user.pdf (Chinese Version)
User Manual Part 2:
I'm sorry that it is unavailable now.

And human-translated documents (W.I.P) are available here now:
https://github.com/loongson-community/docs/tree/master/English-translation-of-Loongson-manual

Both KVM and TCG are available now!

We now already have a full functional Linux kernel (based on Linux-5.4.x
LTS, the kvm host side and guest side have both been upstream for Linux-
5.9, but Linux-5.9 has not been released yet) here:

https://github.com/chenhuacai/linux

Of course the upstream kernel is also usable (though it is "unstable"
now):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

How to use QEMU/Loongson-3?
1, Download kernel source from the above URL;
2, Build a kernel with arch/mips/configs/loongson3_{def,hpc}config;
3, Boot a Loongson-3A4000 host with this kernel (for KVM mode);
4, Build QEMU-master with this patchset;
5, modprobe kvm (only necessary for KVM mode);
6, Use QEMU with TCG:
   qemu-system-mips64el -M loongson3-virt,accel=tcg -cpu Loongson-3A1000 
-kernel  -append ...
   Use QEMU with KVM:
   qemu-system-mips64el -M loongson3-virt,accel=kvm -cpu Loongson-3A4000 
-kernel  -append ...

   The "-cpu" parameter is optional here and QEMU will use the correct type for 
TCG/KVM automatically.

V1 -> V2:
1, Add a cover letter;
2, Improve CPU definitions;
3, Remove LS7A-related things (Use GPEX instead);
4, Add a description of how to run QEMU/Loongson-3.

V2 -> V3:
1, Fix all possible checkpatch.pl errors and warnings.

V3 -> V4:
1, Sync code with upstream;
2, Remove merged patches;
3, Fix build failure without CONFIG_KVM;
4, Add Reviewed-by: Aleksandar Markovic .

V4 -> V5:
1, Improve coding style;
2, Remove merged patches;
3, Rename machine name from "loongson3" to "loongson3-virt";
4, Rework the "loongson3-virt" machine to drop any ISA things;
5, Rework "hw/mips: Implement the kvm_type() hook in MachineClass";
6, Add Jiaxun Yang as a reviewer of Loongson-3.

V5 -> V6:
1, Fix license preamble;
2, Improve commit messages;
3, Add hw/intc/loongson_liointc.c to MAINTAINERS;
4, Fix all possible checkpatch.pl errors and warnings.

V7 and V8 have only one patch (machine definition) with some minor improvements.

V8 -> V9:
1, Update KVM type definition from kernel;
2, Fix PageMask with variable page size for TCG;
3, Add TCG support (add Loongson-EXT instructions).

Huacai Chen and Jiaxun Yang (6):
 linux-headers: Update MIPS KVM type defintition
 target/mips: Fix PageMask with variable page size
 target/mips: Add loongson-ext lswc2 group of instructions (Part 1)
 target/mips: Add loongson-ext lswc2 group of instructions (Part 2)
 target/mips: Add loongson-ext lsdc2 group of instructions
 hw/mips: Add Loongson-3 machine support

Signed-off-by: Huacai Chen 
---
 default-configs/mips64el-softmmu.mak |   1 +
 hw/mips/Kconfig  |  11 +
 hw/mips/fw_cfg.c |  35 ++
 hw/mips/fw_cfg.h |  19 +
 hw/mips/loongson3_virt.c | 956 +++
 hw/mips/meson.build  |   3 +-

[PATCH] target/arm: Fix sve ldr/str

2020-09-15 Thread Richard Henderson

The mte update missed a bit when producing clean addresses.

Fixes: b2aa8879b88
Signed-off-by: Richard Henderson 
---
 target/arm/translate-sve.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c
index e4cd6b6251..c0d8a5863a 100644
--- a/target/arm/translate-sve.c
+++ b/target/arm/translate-sve.c
@@ -4290,7 +4290,7 @@ static void do_ldr(DisasContext *s, uint32_t vofs, int 
len, int rn, int imm)
 for (i = 0; i < len_align; i += 8) {
 tcg_gen_qemu_ld_i64(t0, clean_addr, midx, MO_LEQ);
 tcg_gen_st_i64(t0, cpu_env, vofs + i);
-tcg_gen_addi_i64(clean_addr, cpu_reg_sp(s, rn), 8);
+tcg_gen_addi_i64(clean_addr, clean_addr, 8);
 }
 tcg_temp_free_i64(t0);
 } else {
@@ -4379,7 +4379,7 @@ static void do_str(DisasContext *s, uint32_t vofs, int 
len, int rn, int imm)
 for (i = 0; i < len_align; i += 8) {
 tcg_gen_ld_i64(t0, cpu_env, vofs + i);
 tcg_gen_qemu_st_i64(t0, clean_addr, midx, MO_LEQ);
-tcg_gen_addi_i64(clean_addr, cpu_reg_sp(s, rn), 8);
+tcg_gen_addi_i64(clean_addr, clean_addr, 8);
 }
 tcg_temp_free_i64(t0);
 } else {
-- 
2.25.1

[PATCH v2 2/6] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Richard Henderson

It turns out that some hosts have a default malloc alignment less
than that required for vectors.

We assume that, with compiler annotation on CPUArchState, that we
can properly align the vector portion of the guest state.  Fix the
alignment of the allocation by using qemu_memalloc when required.

Signed-off-by: Richard Henderson 
---
Cc: Paolo Bonzini 
Cc: "Daniel P. Berrangé" 
Cc: Eduardo Habkost 
---
 include/qom/object.h |  5 +
 qom/object.c | 36 +---
 2 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/include/qom/object.h b/include/qom/object.h
index 056f67ab3b..5709c36859 100644
--- a/include/qom/object.h
+++ b/include/qom/object.h
@@ -691,6 +691,7 @@ struct Object
 .parent = TYPE_##PARENT_MODULE_OBJ_NAME, \
 .name = TYPE_##MODULE_OBJ_NAME, \
 .instance_size = sizeof(ModuleObjName), \
+.instance_align = __alignof__(ModuleObjName), \
 .instance_init = module_obj_name##_init, \
 .instance_finalize = module_obj_name##_finalize, \
 .class_size = sizeof(ModuleObjName##Class), \
@@ -770,6 +771,9 @@ struct Object
  * @instance_size: The size of the object (derivative of #Object).  If
  *   @instance_size is 0, then the size of the object will be the size of the
  *   parent object.
+ * @instance_align: The required alignment of the object.  If @instance_align
+ *   is 0, then normal malloc alignment is sufficient; if non-zero, then we
+ *   must use qemu_memalign for allocation.
  * @instance_init: This function is called to initialize an object.  The parent
  *   class will have already been initialized so the type is only responsible
  *   for initializing its own members.
@@ -807,6 +811,7 @@ struct TypeInfo
 const char *parent;
 
 size_t instance_size;
+size_t instance_align;
 void (*instance_init)(Object *obj);
 void (*instance_post_init)(Object *obj);
 void (*instance_finalize)(Object *obj);
diff --git a/qom/object.c b/qom/object.c
index 387efb25eb..fcc15f6d88 100644
--- a/qom/object.c
+++ b/qom/object.c
@@ -50,6 +50,7 @@ struct TypeImpl
 size_t class_size;
 
 size_t instance_size;
+size_t instance_align;
 
 void (*class_init)(ObjectClass *klass, void *data);
 void (*class_base_init)(ObjectClass *klass, void *data);
@@ -114,6 +115,7 @@ static TypeImpl *type_new(const TypeInfo *info)
 
 ti->class_size = info->class_size;
 ti->instance_size = info->instance_size;
+ti->instance_align = info->instance_align;
 
 ti->class_init = info->class_init;
 ti->class_base_init = info->class_base_init;
@@ -688,16 +690,44 @@ static void object_finalize(void *data)
 }
 }
 
+/* Find the minimum alignment guaranteed by the system malloc. */
+#if __STDC_VERSION__ >= 201112L
+typddef max_align_t qemu_max_align_t;
+#else
+typedef union {
+long l;
+void *p;
+double d;
+long double ld;
+} qemu_max_align_t;
+#endif
+
 static Object *object_new_with_type(Type type)
 {
 Object *obj;
+size_t size, align;
+void (*obj_free)(void *);
 
 g_assert(type != NULL);
 type_initialize(type);
 
-obj = g_malloc(type->instance_size);
-object_initialize_with_type(obj, type->instance_size, type);
-obj->free = g_free;
+size = type->instance_size;
+align = type->instance_align;
+
+/*
+ * Do not use qemu_memalign unless required.  Depending on the
+ * implementation, extra alignment implies extra overhead.
+ */
+if (likely(align <= __alignof__(qemu_max_align_t))) {
+obj = g_malloc(size);
+obj_free = g_free;
+} else {
+obj = qemu_memalign(align, size);
+obj_free = qemu_vfree;
+}
+
+object_initialize_with_type(obj, size, type);
+obj->free = obj_free;
 
 return obj;
 }
-- 
2.25.1

[PATCH v2 4/6] target/ppc: Set instance_align on PowerPCCPU TypeInfo

2020-09-15 Thread Richard Henderson

Fix alignment of CPUPPCState.vsr.

Signed-off-by: Richard Henderson 
---
Cc: David Gibson 
Cc: qemu-...@nongnu.org
---
 target/ppc/translate_init.c.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/ppc/translate_init.c.inc b/target/ppc/translate_init.c.inc
index 230a062d29..accb4f2fae 100644
--- a/target/ppc/translate_init.c.inc
+++ b/target/ppc/translate_init.c.inc
@@ -10960,6 +10960,7 @@ static const TypeInfo ppc_cpu_type_info = {
 .name = TYPE_POWERPC_CPU,
 .parent = TYPE_CPU,
 .instance_size = sizeof(PowerPCCPU),
+.instance_align = __alignof__(PowerPCCPU),
 .instance_init = ppc_cpu_instance_init,
 .instance_finalize = ppc_cpu_instance_finalize,
 .abstract = true,
-- 
2.25.1

[PATCH v2 6/6] target/s390x: Set instance_align on S390CPU TypeInfo

2020-09-15 Thread Richard Henderson

Fix alignment of CPUS390XState.vregs.

Signed-off-by: Richard Henderson 
---
Cc: David Hildenbrand 
Cc: Cornelia Huck 
Cc: qemu-s3...@nongnu.org
---
 target/s390x/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c
index 749cd548f0..e350edc9f5 100644
--- a/target/s390x/cpu.c
+++ b/target/s390x/cpu.c
@@ -517,6 +517,7 @@ static const TypeInfo s390_cpu_type_info = {
 .name = TYPE_S390_CPU,
 .parent = TYPE_CPU,
 .instance_size = sizeof(S390CPU),
+.instance_align = __alignof__(S390CPU),
 .instance_init = s390_cpu_initfn,
 .instance_finalize = s390_cpu_finalize,
 .abstract = true,
-- 
2.25.1

[PATCH v2 1/6] util/oslib-win32: Use _aligned_malloc for qemu_try_memalign

2020-09-15 Thread Richard Henderson

We do not need or want to be allocating page sized quanta.

Signed-off-by: Richard Henderson 
---
Cc: Stefan Weil 
---
 util/oslib-win32.c | 10 +++---
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/util/oslib-win32.c b/util/oslib-win32.c
index c654dafd93..8d838bf342 100644
--- a/util/oslib-win32.c
+++ b/util/oslib-win32.c
@@ -56,10 +56,8 @@ void *qemu_try_memalign(size_t alignment, size_t size)
 {
 void *ptr;
 
-if (!size) {
-abort();
-}
-ptr = VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_READWRITE);
+assert(size != 0);
+ptr = _aligned_malloc(alignment, size);
 trace_qemu_memalign(alignment, size, ptr);
 return ptr;
 }
@@ -93,9 +91,7 @@ void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool 
shared)
 void qemu_vfree(void *ptr)
 {
 trace_qemu_vfree(ptr);
-if (ptr) {
-VirtualFree(ptr, 0, MEM_RELEASE);
-}
+_aligned_free(ptr);
 }
 
 void qemu_anon_ram_free(void *ptr, size_t size)
-- 
2.25.1

[PATCH v2 5/6] target/riscv: Set instance_align on RISCVCPU TypeInfo

2020-09-15 Thread Richard Henderson

Fix alignment of CPURISCVState.vreg.

Signed-off-by: Richard Henderson 
---
Cc: Alistair Francis 
Cc: qemu-ri...@nongnu.org
---
 target/riscv/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 57c006df5d..0bbfd7f457 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -628,6 +628,7 @@ static const TypeInfo riscv_cpu_type_infos[] = {
 .name = TYPE_RISCV_CPU,
 .parent = TYPE_CPU,
 .instance_size = sizeof(RISCVCPU),
+.instance_align = __alignof__(RISCVCPU),
 .instance_init = riscv_cpu_init,
 .abstract = true,
 .class_size = sizeof(RISCVCPUClass),
-- 
2.25.1

[PATCH v2 3/6] target/arm: Set instance_align on CPUARM TypeInfo

2020-09-15 Thread Richard Henderson

Fix alignment of CPUARMState.vfp.zregs.

Signed-off-by: Richard Henderson 
---
Cc: Peter Maydell 
Cc: qemu-...@nongnu.org
---
 target/arm/cpu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 7b5ea65fab..a7643deab4 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -2290,6 +2290,7 @@ void arm_cpu_register(const ARMCPUInfo *info)
 TypeInfo type_info = {
 .parent = TYPE_ARM_CPU,
 .instance_size = sizeof(ARMCPU),
+.instance_align = __alignof__(ARMCPU),
 .instance_init = arm_cpu_instance_init,
 .class_size = sizeof(ARMCPUClass),
 .class_init = info->class_init ?: cpu_register_class_init,
@@ -2305,6 +2306,7 @@ static const TypeInfo arm_cpu_type_info = {
 .name = TYPE_ARM_CPU,
 .parent = TYPE_CPU,
 .instance_size = sizeof(ARMCPU),
+.instance_align = __alignof__(ARMCPU),
 .instance_init = arm_cpu_initfn,
 .instance_finalize = arm_cpu_finalizefn,
 .abstract = true,
-- 
2.25.1

[PATCH v2 0/6] qom: Allow object to be aligned

2020-09-15 Thread Richard Henderson

I've seen some failures on arm and s390x hosts after
enabling host vector support.  It turns out that the
malloc for these hosts does not provide 16-byte alignment.

We already have a function that can alloc with alignment,
but we need to pass this down from the structure.  We also
don't want to use this function unconditionally, because
the windows version does page allocation, which would be
overkill for the vast majority of the objects allocated.

Changes in v2:
  * Add _aligned_malloc patch for win32.  For what it's
worth, this passes a gitlab cross-compile test.

  * Add and use qemu_max_align_t for choosing between
g_malloc and qemu_memalign.

I had been discussing extra checks for i386-linux with
Eduardo, but then it occured to me that both linux libc
posix_memalign is smart enough to not imply extra overhead.
So qemu_memalign with alignment <= malloc alignment is
handled easily by the system.


r~


Cc: Paolo Bonzini 
Cc: "Daniel P. Berrangé" 
Cc: Eduardo Habkost 
Cc: Peter Maydell 
Cc: David Gibson 
Cc: Alistair Francis 
Cc: David Hildenbrand 
Cc: Cornelia Huck 
Cc: qemu-...@nongnu.org
Cc: qemu-...@nongnu.org
Cc: qemu-ri...@nongnu.org
Cc: qemu-s3...@nongnu.org


Richard Henderson (6):
  util/oslib-win32: Use _aligned_malloc for qemu_try_memalign
  qom: Allow objects to be allocated with increased alignment
  target/arm: Set instance_align on CPUARM TypeInfo
  target/ppc: Set instance_align on PowerPCCPU TypeInfo
  target/riscv: Set instance_align on RISCVCPU TypeInfo
  target/s390x: Set instance_align on S390CPU TypeInfo

 include/qom/object.h|  5 +
 qom/object.c| 36 ++---
 target/arm/cpu.c|  2 ++
 target/riscv/cpu.c  |  1 +
 target/s390x/cpu.c  |  1 +
 util/oslib-win32.c  | 10 +++--
 target/ppc/translate_init.c.inc |  1 +
 7 files changed, 46 insertions(+), 10 deletions(-)

-- 
2.25.1

[Bug 1894869] Re: Chelsio T4 has old MSIX PBA offset bug

2020-09-15 Thread Nick Bauer

I was able to boot a VM with just the functions of the device with the
ethernet controller function ID added as PCI devices. Something I
noticed while adding in those devices though is that all of the others
have a description associated with them in Proxmox, but the one that's
causing the boot to fail doesn't. I attached a picture of the menu,
81:00.7 has no functions associated with it. So it seems like it just
doesn't have any function at all? Unless it benefits QEMU to know
whether turning SR-IOV on for these cards fixes the problem, I don't
think I'm going to go through the process of turning  it on, since the
process looks terrible. Thank you for your help.

** Attachment added: "no 7th.JPG"
   
https://bugs.launchpad.net/qemu/+bug/1894869/+attachment/5411172/+files/no%207th.JPG

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894869

Title:
  Chelsio T4 has old MSIX PBA offset bug

Status in QEMU:
  New
Status in Debian:
  In Progress

Bug description:
  There exists a bug with Chelsio NICs T4 that causes the following
  error:

  kvm: -device vfio-
  pci,host=:83:00.7,id=hostpci1.7,bus=pci.0,addr=0x11.7: vfio
  :83:00.7: hardware reports invalid configuration, MSIX PBA outside
  of specified BAR

  I discovered this bug on a Proxmox system, and I was working with a
  downstream Proxmox developer to try to fix this issue. They provided
  me with the following change to make from line 1484 of hw/vfio/pci.c:

  static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp)
    * is 0x1000, so we hard code that here.
    */
   if (vdev->vendor_id == PCI_VENDOR_ID_CHELSIO &&
  -(vdev->device_id & 0xff00) == 0x5800) {
  +((vdev->device_id & 0xff00) == 0x5800 ||
  + (vdev->device_id & 0xff00) == 0x1425)) {
   msix->pba_offset = 0x1000;
   } else if (vdev->msix_relo == OFF_AUTOPCIBAR_OFF) {
   error_setg(errp, "hardware reports invalid configuration, "

  However, I found that this did not fix the issue, so the bug appears
  to work differently than the one that was present on the T5 NICs which
  has already been patched. I have attached the output of my lspci
  -nnkvv

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1894869/+subscriptions

[Bug 1892604] Re: qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion `addr <= GINTSTS2' failed.

2020-09-15 Thread Paul Zimmerman

Hmm, that's odd. I will try to reproduce this.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1892604

Title:
  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

Status in QEMU:
  New

Bug description:
  When trying to run the 2016-05-27 Raspbian image on the emulated
  raspi2 platform, the system boots but shortly after the login prompt
  QEMU (master; commit ID ca489cd037e4d50dc6c40570a167504ad7e5a521) dies
  with:

  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

  Steps to reproduce:

  1. Get the image: wget
  
http://downloads.raspberrypi.org/raspbian/images/raspbian-2016-05-31/2016-05-27
  -raspbian-jessie.zip

  2. Extract the kernel image and DTB:

  sudo losetup -f --show -P 2016-05-27-raspbian-jessie.img
  sudo mkdir /mnt/rpi
  sudo mount /dev/loop11p1 /mnt/rpi/
  cp /mnt/rpi/kernel7.img . 



  cp /mnt/rpi/bcm2709-rpi-2-b.dtb . 



  sudo umount /mnt/rpi 
  sudo losetup -d /dev/loop11 

  3. Run QEMU:
  qemu-system-arm -M raspi2 -m 1G -dtb bcm2709-rpi-2-b.dtb -kernel kernel7.img 
-append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 
root=/dev/mmcblk0p2" -sd 2016-05-27-raspbian-jessie.img -smp 4 -serial stdio 
-display none

  A few seconds after the login prompt is displayed, QEMU will exit with
  the assertion failure.

  I also tried changing all of the asserts to if statements that (for
  MMIO reads) returned 0 and (for writes) just returned, but this
  resulted in a non-responsive system.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1892604/+subscriptions

[PATCH 33/37] qapi/introspect.py: add notational type hints

2020-09-15 Thread John Snow

Add a tiny hint into schema.py to allow introspect.py to type check, too.

Signed-off-by: John Snow 
---
 scripts/qapi/introspect.py | 123 ++---
 scripts/qapi/mypy.ini  |   5 --
 scripts/qapi/schema.py |   2 +-
 3 files changed, 87 insertions(+), 43 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index e0f5007ab7..4e1ea3a00c 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,16 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, List, Optional, Sequence)
+from typing import (
+Any,
+Dict,
+Generic,
+List,
+NamedTuple,
+Optional,
+Sequence,
+TypeVar,
+)
 
 from .common import (
 c_name,
@@ -19,8 +28,23 @@
 mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaFeature, QAPISchemaType)
+from .schema import (
+QAPISchema,
+QAPISchemaArrayType,
+QAPISchemaBuiltinType,
+QAPISchemaEntity,
+QAPISchemaEnumMember,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaObjectTypeMember,
+QAPISchemaType,
+QAPISchemaVariant,
+QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+
+_NodeType = TypeVar('_NodeType')
 
 
 class Extra(NamedTuple):
@@ -31,23 +55,24 @@ class Extra(NamedTuple):
 ifcond: Sequence[str] = tuple()
 
 
-class Node:
+class Node(Generic[_NodeType]):
 """
 Node generally contains a SchemaInfo-like type (as a dict),
 But it also used to wrap comments/ifconds around leaf value types.
 """
 # Remove after 3.7 adds @dataclass:
 # pylint: disable=too-few-public-methods
-def __init__(self, data, ifcond: List[str],
+def __init__(self, data: _NodeType, ifcond: List[str],
  extra: Optional[Extra] = None):
 self.data = data
 comment = extra.comment if extra else None
 self.extra = Extra(comment, ifcond)
 
 
-def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
+def _tree_to_qlit(obj: Any, level: int = 0,
+  suppress_first_indent: bool = False) -> str:
 
-def indent(level):
+def indent(level: int) -> str:
 return level * 4 * ' '
 
 if isinstance(obj, Node):
@@ -94,21 +119,20 @@ def indent(level):
 return ret
 
 
-def to_c_string(string):
+def to_c_string(string: str) -> str:
 return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
-
-def __init__(self, prefix, unmask):
+def __init__(self, prefix: str, unmask: bool):
 super().__init__(
 prefix, 'qapi-introspect',
 ' * QAPI/QMP schema introspection', __doc__)
 self._unmask = unmask
-self._schema = None
-self._trees = []
-self._used_types = []
-self._name_map = {}
+self._schema: Optional[QAPISchema] = None
+self._trees: List[Node[Dict[str, Any]]] = []
+self._used_types: List[QAPISchemaType] = []
+self._name_map: Dict[str, str] = {}
 self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqapi-introspect.h"
@@ -116,10 +140,10 @@ def __init__(self, prefix, unmask):
 ''',
  prefix=prefix))
 
-def visit_begin(self, schema):
+def visit_begin(self, schema: QAPISchema) -> None:
 self._schema = schema
 
-def visit_end(self):
+def visit_end(self) -> None:
 # visit the types that are actually used
 for typ in self._used_types:
 typ.visit(self)
@@ -141,18 +165,18 @@ def visit_end(self):
 self._used_types = []
 self._name_map = {}
 
-def visit_needed(self, entity):
+def visit_needed(self, entity: QAPISchemaEntity) -> bool:
 # Ignore types on first pass; visit_end() will pick up used types
 return not isinstance(entity, QAPISchemaType)
 
-def _name(self, name):
+def _name(self, name: str) -> str:
 if self._unmask:
 return name
 if name not in self._name_map:
 self._name_map[name] = '%d' % len(self._name_map)
 return self._name_map[name]
 
-def _use_type(self, typ):
+def _use_type(self, typ: QAPISchemaType) -> str:
 # Map the various integer types to plain int
 if typ.json_type() == 'int':
 typ = self._schema.lookup_type('int')
@@ -172,10 +196,13 @@ def _use_type(self, typ):
 return self._name(typ.name)
 
 @classmethod
-def _gen_features(cls, features: List[QAPISchemaFeature]):
+def _gen_features(cls,
+  features: List[QAPISchemaFeature]) -> List[Node[str]]:
 return [Node(f.name, f.ifcond) for f in features]
 
-def _gen_tree(self, name, mtype, obj, ifcond, features):
+def _gen_tree(self, name: str, mtype: str, obj: Dict[str, Any],
+

[PATCH 35/37] qapi/types.py: remove one-letter variables

2020-09-15 Thread John Snow

"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

Signed-off-by: John Snow 
---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++--
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 3aebb0d228..9ca6b60caf 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
 expr.py,
 parser.py,
 schema.py,
-types.py,
 visit.py,
 
 
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 274d62fac8..a0a06b68c5 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
 .array = (const char *const[]) {
 ''',
 c_name=c_name(name))
-for m in members:
-ret += gen_if(m.ifcond)
-index = c_enum_const(name, m.name, prefix)
+for member in members:
+ret += gen_if(member.ifcond)
+index = c_enum_const(name, member.name, prefix)
 ret += mcgen('''
 [%(index)s] = "%(name)s",
 ''',
- index=index, name=m.name)
-ret += gen_endif(m.ifcond)
+ index=index, name=member.name)
+ret += gen_endif(member.ifcond)
 
 ret += mcgen('''
 },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
 c_name=c_name(name))
 
-for m in enum_members:
-ret += gen_if(m.ifcond)
+for member in enum_members:
+ret += gen_if(member.ifcond)
 ret += mcgen('''
 %(c_enum)s,
 ''',
- c_enum=c_enum_const(name, m.name, prefix))
-ret += gen_endif(m.ifcond)
+ c_enum=c_enum_const(name, member.name, prefix))
+ret += gen_endif(member.ifcond)
 
 ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
 objects_seen.add(name)
 
 ret = ''
-if variants:
-for v in variants.variants:
-if isinstance(v.type, QAPISchemaObjectType):
-ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-  v.type.local_members, v.type.variants)
+for variant in variants.variants if variants else ():
+obj = variant.type
+if not isinstance(obj, QAPISchemaObjectType):
+continue
+ret += gen_object(obj.name, obj.ifcond, obj.base,
+  obj.local_members, obj.variants)
 
 ret += mcgen('''
 
-- 
2.26.2

Re: [PATCH 0/5] qom: Allow object to be aligned

2020-09-15 Thread Richard Henderson

On 9/15/20 10:46 AM, Richard Henderson wrote:
> We already have a function that can alloc with alignment,
> but we need to pass this down from the structure.  We also
> don't want to use this function unconditionally, because
> the windows version does page allocation, which would be
> overkill for the vast majority of the objects allocated.

Stefan, why are we using VirtualAlloc in util/oslib-win32.c, qemu_try_memalign,
instead of _aligned_malloc?


r~

[PATCH 31/37] qapi/introspect.py: add _gen_features helper

2020-09-15 Thread John Snow

_make_tree doesn't know if it is receiving an object or some other type;
adding features information should arguably be performed by the caller.

This will help us refactor _make_tree more gracefully in the next patch.

Signed-off-by: John Snow 
---
 scripts/qapi/introspect.py | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 41ca8afc67..e1edd0b179 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,7 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, Optional, Sequence)
+from typing import (NamedTuple, List, Optional, Sequence)
 
 from .common import (
 c_name,
@@ -20,7 +20,7 @@
 )
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaType)
+ QAPISchemaFeature, QAPISchemaType)
 
 
 class Extra(NamedTuple):
@@ -31,12 +31,10 @@ class Extra(NamedTuple):
 ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond, features,
+def _make_tree(obj, ifcond,
extra: Optional[Extra] = None):
 comment = extra.comment if extra else None
 extra = Extra(comment, ifcond)
-if features:
-obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
 return (obj, extra)
 
 
@@ -169,6 +167,10 @@ def _use_type(self, typ):
 return '[' + self._use_type(typ.element_type) + ']'
 return self._name(typ.name)
 
+@classmethod
+def _gen_features(cls, features: List[QAPISchemaFeature]):
+return [_make_tree(f.name, f.ifcond) for f in features]
+
 def _gen_tree(self, name, mtype, obj, ifcond, features):
 extra = None
 if mtype not in ('command', 'event', 'builtin', 'array'):
@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
 name = self._name(name)
 obj['name'] = name
 obj['meta-type'] = mtype
-self._trees.append(_make_tree(obj, ifcond, features, extra))
+if features:
+obj['features'] = self._gen_features(features)
+self._trees.append(_make_tree(obj, ifcond, extra))
 
 def _gen_member(self, member):
 obj = {'name': member.name, 'type': self._use_type(member.type)}
 if member.optional:
 obj['default'] = None
-return _make_tree(obj, member.ifcond, member.features)
+if member.features:
+obj['features'] = self._gen_features(member.features)
+return _make_tree(obj, member.ifcond)
 
 def _gen_variants(self, tag_name, variants):
 return {'tag': tag_name,
@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants):
 
 def _gen_variant(self, variant):
 obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-return _make_tree(obj, variant.ifcond, None)
+return _make_tree(obj, variant.ifcond)
 
 def visit_builtin_type(self, name, info, json_type):
 self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
-- 
2.26.2

[PATCH 25/37] qapi/gen.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/gen.py | 105 +++-
 1 file changed, 56 insertions(+), 49 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fc46813d1a..c87c84e7c9 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -15,6 +15,7 @@
 import errno
 import os
 import re
+from typing import Dict, Generator, List, Optional, Tuple
 
 from .common import (
 c_fname,
@@ -24,32 +25,34 @@
 guardstart,
 mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import (
+QAPISchemaVisitor,
+)
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-def __init__(self, fname):
+def __init__(self, fname: Optional[str]):
 self.fname = fname
 self._preamble = ''
 self._body = ''
 
-def preamble_add(self, text):
+def preamble_add(self, text: str) -> None:
 self._preamble += text
 
-def add(self, text):
+def add(self, text: str) -> None:
 self._body += text
 
-def get_content(self):
+def get_content(self) -> str:
 return self._top() + self._preamble + self._body + self._bottom()
 
-def _top(self):
+def _top(self) -> str:
 return ''
 
-def _bottom(self):
+def _bottom(self) -> str:
 return ''
 
-def write(self, output_dir):
+def write(self, output_dir: str) -> None:
 # Include paths starting with ../ are used to reuse modules of the main
 # schema in specialised schemas. Don't overwrite the files that are
 # already generated for the main schema.
@@ -74,7 +77,7 @@ def write(self, output_dir):
 f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
 if before == after:
 return after   # suppress empty #if ... #endif
 
@@ -91,40 +94,38 @@ def _wrap_ifcond(ifcond, before, after):
 
 
 class QAPIGenCCode(QAPIGen):
-
-def __init__(self, fname):
+def __init__(self, fname: Optional[str]):
 super().__init__(fname)
-self._start_if = None
+self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-def start_if(self, ifcond):
+def start_if(self, ifcond: List[str]) -> None:
 assert self._start_if is None
 self._start_if = (ifcond, self._body, self._preamble)
 
-def end_if(self):
+def end_if(self) -> None:
 assert self._start_if
 self._wrap_ifcond()
 self._start_if = None
 
-def _wrap_ifcond(self):
+def _wrap_ifcond(self) -> None:
 self._body = _wrap_ifcond(self._start_if[0],
   self._start_if[1], self._body)
 self._preamble = _wrap_ifcond(self._start_if[0],
   self._start_if[2], self._preamble)
 
-def get_content(self):
+def get_content(self) -> str:
 assert self._start_if is None
 return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-def __init__(self, fname, blurb, pydoc):
+def __init__(self, fname: str, blurb: str, pydoc: str):
 super().__init__(fname)
 self._blurb = blurb
 self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
   re.MULTILINE))
 
-def _top(self):
+def _top(self) -> str:
 return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -140,7 +141,7 @@ def _top(self):
 ''',
  blurb=self._blurb, copyright=self._copyright)
 
-def _bottom(self):
+def _bottom(self) -> str:
 return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -150,16 +151,16 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-def _top(self):
+def _top(self) -> str:
 return super()._top() + guardstart(self.fname)
 
-def _bottom(self):
+def _bottom(self) -> str:
 return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str],
+  *args: QAPIGenCCode) -> Generator[None, None, None]:
 """A 'with' statement context manager to wrap with start_if()/end_if()
 
 *args: any number of QAPIGenCCode
@@ -185,15 +186,17 @@ def ifcontext(ifcond, *args):
 
 
 class QAPIGenDoc(QAPIGen):
-
-def _top(self):
+def _top(self) -> str:
 return (super()._top()
 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-def __init__(self, prefix, what, blurb, pydoc):
+def __init__(self,
+ prefix: str,
+ what: str,
+ blurb: str,
+ pydoc: str):
 self._prefix = prefix
 self._what = what
 self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -201,38 +204,42 @@ def __init__(self, prefix, what, blurb, pydoc):
 self._genh = QAPIGenH(self._prefix + self._what + '.h',

[PATCH 36/37] qapi/visit.py: remove unused parameters from gen_visit_object

2020-09-15 Thread John Snow

And this fixes the pylint report for this file, so make sure we check
this in the future, too.

Signed-off-by: John Snow 
---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 9ca6b60caf..9d7a1d285f 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
 expr.py,
 parser.py,
 schema.py,
-visit.py,
 
 
 [MESSAGES CONTROL]
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 66ce3dc52e..ce71659a03 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -247,7 +247,7 @@ def gen_visit_alternate(name, variants):
 return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -340,7 +340,7 @@ def visit_object_type(self, name, info, ifcond, features,
 if not name.startswith('q_'):
 # only explicit types need an allocating visit
 self._genh.add(gen_visit_decl(name))
-self._genc.add(gen_visit_object(name, base, members, variants))
+self._genc.add(gen_visit_object(name))
 
 def visit_alternate_type(self, name, info, ifcond, features, variants):
 with ifcontext(ifcond, self._genh, self._genc):
-- 
2.26.2

[PATCH 26/37] qapi/gen.py: Enable checking with mypy

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/mypy.ini | 5 -
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 43c8bd1973..dbfeda748c 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.introspect]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH 27/37] qapi/gen.py: Remove unused parameter

2020-09-15 Thread John Snow

module_basename doesn't use the 'what' argument, so remove it.

Signed-off-by: John Snow 
---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index c87c84e7c9..e873901672 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -234,7 +234,7 @@ def _is_user_module(name: Optional[str]) -> bool:
 def _is_builtin_module(name: Optional[str]) -> bool:
 return not name
 
-def _module_dirname(self, what: str, name: Optional[str]) -> str:
+def _module_dirname(self, name: Optional[str]) -> str:
 if self._is_user_module(name):
 return os.path.dirname(name)
 return ''
@@ -252,7 +252,7 @@ def _module_basename(self, what: str, name: Optional[str]) 
-> str:
 return ret
 
 def _module_filename(self, what: str, name: Optional[str]) -> str:
-return os.path.join(self._module_dirname(what, name),
+return os.path.join(self._module_dirname(name),
 self._module_basename(what, name))
 
 def _add_module(self, name: Optional[str], blurb: str) -> None:
-- 
2.26.2

[PATCH 20/37] qapi/commands.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/commands.py | 72 ++--
 1 file changed, 54 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index b215e58357..3c950da10a 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,15 +13,32 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import Dict, Optional, List, Set
+
 from .common import (
 c_name,
 mcgen,
 )
+from .gen import (
+QAPIGenC,
+QAPIGenCCode,
+QAPISchemaModularCVisitor,
+ifcontext,
+)
 from .params import build_params
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .schema import (
+QAPISchema,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
 return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -30,7 +47,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
 ret = ''
 
 argstr = ''
@@ -66,7 +86,7 @@ def gen_call(name, arg_type, boxed, ret_type):
 return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
 return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
@@ -87,19 +107,22 @@ def gen_marshal_output(ret_type):
  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
 return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
 % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
 return mcgen('''
 %(proto)s;
 ''',
  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+arg_type: Optional[QAPISchemaObjectType],
+boxed: bool,
+ret_type: Optional[QAPISchemaType]) -> str:
 have_args = boxed or (arg_type and not arg_type.is_empty())
 
 ret = mcgen('''
@@ -181,7 +204,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
 return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+ success_response: bool,
+ allow_oob: bool,
+ allow_preconfig: bool) -> str:
 options = []
 
 if not success_response:
@@ -203,7 +229,7 @@ def gen_register_command(name, success_response, allow_oob, 
allow_preconfig):
 return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
 ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -220,15 +246,14 @@ def gen_registry(registry, prefix):
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-commands',
 ' * Schema-defined QAPI/QMP commands', None, __doc__)
 self._regy = QAPIGenCCode(None)
-self._visited_ret_types = {}
+self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 self._visited_ret_types[self._genc] = set()
 commands = self._module_basename('qapi-commands', name)
 types = self._module_basename('qapi-types', name)
@@ -252,7 +277,7 @@ def _begin_user_module(self, name):
 ''',
  types=types))
 
-def visit_end(self):
+def visit_end(self) -> None:
 self._add_system_module('init', ' * QAPI Commands initialization')
 self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -268,9 +293,18 @@ def visit_end(self):
   prefix=self._prefix))
 self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
 
-def visit_command(self, name, info, ifcond, features,
-  arg_type, ret_type, gen, success_response, boxed,
-  allow_oob, allow_preconfig):
+def visit_command(self,
+  name: str,
+  info: QAPISourceInfo,
+  ifcond: List[str],
+  features: List[QAPISchemaFeature],
+

[PATCH 32/37] qapi/introspect.py: create a typed 'Node' data structure

2020-09-15 Thread John Snow

Replacing the un-typed tuple, add a typed Node that we can add typed
metadata to.

Signed-off-by: John Snow 
---
 scripts/qapi/introspect.py | 53 --
 1 file changed, 28 insertions(+), 25 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index e1edd0b179..e0f5007ab7 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -31,11 +31,18 @@ class Extra(NamedTuple):
 ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond,
-   extra: Optional[Extra] = None):
-comment = extra.comment if extra else None
-extra = Extra(comment, ifcond)
-return (obj, extra)
+class Node:
+"""
+Node generally contains a SchemaInfo-like type (as a dict),
+But it also used to wrap comments/ifconds around leaf value types.
+"""
+# Remove after 3.7 adds @dataclass:
+# pylint: disable=too-few-public-methods
+def __init__(self, data, ifcond: List[str],
+ extra: Optional[Extra] = None):
+self.data = data
+comment = extra.comment if extra else None
+self.extra = Extra(comment, ifcond)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -43,18 +50,15 @@ def _tree_to_qlit(obj, level=0, 
suppress_first_indent=False):
 def indent(level):
 return level * 4 * ' '
 
-if isinstance(obj, tuple):
-ifobj, extra = obj
-ifcond = extra.ifcond
-comment = extra.comment
+if isinstance(obj, Node):
 ret = ''
-if comment:
-ret += indent(level) + '/* %s */\n' % comment
-if ifcond:
-ret += gen_if(ifcond)
-ret += _tree_to_qlit(ifobj, level)
-if ifcond:
-ret += '\n' + gen_endif(ifcond)
+if obj.extra.comment:
+ret += indent(level) + '/* %s */\n' % obj.extra.comment
+if obj.extra.ifcond:
+ret += gen_if(obj.extra.ifcond)
+ret += _tree_to_qlit(obj.data, level)
+if obj.extra.ifcond:
+ret += '\n' + gen_endif(obj.extra.ifcond)
 return ret
 
 ret = ''
@@ -169,7 +173,7 @@ def _use_type(self, typ):
 
 @classmethod
 def _gen_features(cls, features: List[QAPISchemaFeature]):
-return [_make_tree(f.name, f.ifcond) for f in features]
+return [Node(f.name, f.ifcond) for f in features]
 
 def _gen_tree(self, name, mtype, obj, ifcond, features):
 extra = None
@@ -183,7 +187,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
 obj['meta-type'] = mtype
 if features:
 obj['features'] = self._gen_features(features)
-self._trees.append(_make_tree(obj, ifcond, extra))
+self._trees.append(Node(obj, ifcond, extra))
 
 def _gen_member(self, member):
 obj = {'name': member.name, 'type': self._use_type(member.type)}
@@ -191,7 +195,7 @@ def _gen_member(self, member):
 obj['default'] = None
 if member.features:
 obj['features'] = self._gen_features(member.features)
-return _make_tree(obj, member.ifcond)
+return Node(obj, member.ifcond)
 
 def _gen_variants(self, tag_name, variants):
 return {'tag': tag_name,
@@ -199,15 +203,14 @@ def _gen_variants(self, tag_name, variants):
 
 def _gen_variant(self, variant):
 obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-return _make_tree(obj, variant.ifcond)
+return Node(obj, variant.ifcond)
 
 def visit_builtin_type(self, name, info, json_type):
 self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
 
 def visit_enum_type(self, name, info, ifcond, features, members, prefix):
 self._gen_tree(name, 'enum',
-   {'values': [_make_tree(m.name, m.ifcond, None)
-   for m in members]},
+   {'values': [Node(m.name, m.ifcond) for m in members]},
ifcond, features)
 
 def visit_array_type(self, name, info, ifcond, element_type):
@@ -227,9 +230,9 @@ def visit_object_type_flat(self, name, info, ifcond, 
features,
 def visit_alternate_type(self, name, info, ifcond, features, variants):
 self._gen_tree(name, 'alternate',
{'members': [
-   _make_tree({'type': self._use_type(m.type)},
-  m.ifcond, None)
-   for m in variants.variants]},
+   Node({'type': self._use_type(m.type)}, m.ifcond)
+   for m in variants.variants
+   ]},
ifcond, features)
 
 def visit_command(self, name, info, ifcond, features,
-- 
2.26.2

[PATCH 30/37] qapi/introspect.py: Add a typed 'extra' structure

2020-09-15 Thread John Snow

Typing arbitrarily shaped dicts with mypy is difficult prior to Python
3.8; using explicit structures is nicer.

Since we always define an Extra type now, the return type of _make_tree
simplifies and always returns the tuple.

Signed-off-by: John Snow 
---
 scripts/qapi/introspect.py | 31 +++
 1 file changed, 19 insertions(+), 12 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index b036fcf9ce..41ca8afc67 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,6 +10,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import (NamedTuple, Optional, Sequence)
+
 from .common import (
 c_name,
 gen_endif,
@@ -21,16 +23,21 @@
  QAPISchemaType)
 
 
-def _make_tree(obj, ifcond, features, extra=None):
-if extra is None:
-extra = {}
-if ifcond:
-extra['if'] = ifcond
+class Extra(NamedTuple):
+"""
+Extra contains data that isn't intended for output by introspection.
+"""
+comment: Optional[str] = None
+ifcond: Sequence[str] = tuple()
+
+
+def _make_tree(obj, ifcond, features,
+   extra: Optional[Extra] = None):
+comment = extra.comment if extra else None
+extra = Extra(comment, ifcond)
 if features:
-obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
-if extra:
-return (obj, extra)
-return obj
+obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
+return (obj, extra)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -40,8 +47,8 @@ def indent(level):
 
 if isinstance(obj, tuple):
 ifobj, extra = obj
-ifcond = extra.get('if')
-comment = extra.get('comment')
+ifcond = extra.ifcond
+comment = extra.comment
 ret = ''
 if comment:
 ret += indent(level) + '/* %s */\n' % comment
@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
 if not self._unmask:
 # Output a comment to make it easy to map masked names
 # back to the source when reading the generated output.
-extra = {'comment': '"%s" = %s' % (self._name(name), name)}
+extra = Extra(comment=f'"{self._name(name)}" = {name}')
 name = self._name(name)
 obj['name'] = name
 obj['meta-type'] = mtype
-- 
2.26.2

[PATCH 17/37] qapi/events.py: add notational type hints

2020-09-15 Thread John Snow

This is enough to enable type-checking on events.py, so enable that in
this patch, too.

Signed-off-by: John Snow 
---
 scripts/qapi/events.py | 46 --
 scripts/qapi/mypy.ini  |  5 -
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 75eda72534..449a700ffe 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,6 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import (
 c_enum_const,
 c_name,
@@ -19,17 +21,27 @@
 )
 from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+   arg_type: QAPISchemaObjectType,
+   boxed: bool) -> str:
 return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
 'c_name': c_name(name.lower()),
 'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+arg_type: QAPISchemaObjectType,
+boxed: bool) -> str:
 return mcgen('''
 
 %(proto)s;
@@ -38,7 +50,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
 assert not typ.variants
 ret = mcgen('''
 %(c_name)s param = {
@@ -66,7 +78,11 @@ def gen_param_var(typ):
 return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+   arg_type: QAPISchemaObjectType,
+   boxed: bool,
+   event_enum_name: str,
+   event_emit: str) -> str:
 # FIXME: Our declaration of local variables (and of 'errp' in the
 # parameter list) can collide with exploded members of the event's
 # data type passed in as parameters.  If this collision ever hits in
@@ -142,15 +158,15 @@ def gen_event_send(name, arg_type, boxed, 
event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-events',
 ' * Schema-defined QAPI/QMP events', None, __doc__)
 self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-self._event_enum_members = []
+self._event_enum_members: List[QAPISchemaEnumMember] = []
 self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 events = self._module_basename('qapi-events', name)
 types = self._module_basename('qapi-types', name)
 visit = self._module_basename('qapi-visit', name)
@@ -173,7 +189,7 @@ def _begin_user_module(self, name):
 ''',
  types=types))
 
-def visit_end(self):
+def visit_end(self) -> None:
 self._add_system_module('emit', ' * QAPI Events emission')
 self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -194,7 +210,13 @@ def visit_end(self):
  event_emit=self._event_emit_name,
  event_enum=self._event_enum_name))
 
-def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+def visit_event(self,
+name: str,
+info: QAPISourceInfo,
+ifcond: List[str],
+features: List[QAPISchemaFeature],
+arg_type: QAPISchemaObjectType,
+boxed: bool) -> None:
 with ifcontext(ifcond, self._genh, self._genc):
 self._genh.add(gen_event_send_decl(name, arg_type, boxed))
 self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -205,7 +227,9 @@ def visit_event(self, name, info, ifcond, features, 
arg_type, boxed):
 self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+   output_dir: str,
+   prefix: str) -> None:
 vis = QAPISchemaGenEventVisitor(prefix)
 schema.visit(vis)
 vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index a0f2365a53..b668776d94 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False

[PATCH 28/37] qapi/gen.py: update write() to be more idiomatic

2020-09-15 Thread John Snow

Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.

Signed-off-by: John Snow 
---
 scripts/qapi/gen.py | 24 ++--
 1 file changed, 10 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index e873901672..bd67f6b069 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -12,7 +12,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import Dict, Generator, List, Optional, Tuple
@@ -60,21 +59,18 @@ def write(self, output_dir: str) -> None:
 return
 pathname = os.path.join(output_dir, self.fname)
 odir = os.path.dirname(pathname)
+
 if odir:
-try:
-os.makedirs(odir)
-except os.error as e:
-if e.errno != errno.EEXIST:
-raise
+os.makedirs(odir, exist_ok=True)
+
 fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-f = open(fd, 'r+', encoding='utf-8')
-text = self.get_content()
-oldtext = f.read(len(text) + 1)
-if text != oldtext:
-f.seek(0)
-f.truncate(0)
-f.write(text)
-f.close()
+with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+text = self.get_content()
+oldtext = fp.read(len(text) + 1)
+if text != oldtext:
+fp.seek(0)
+fp.truncate(0)
+fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
-- 
2.26.2

[PATCH 21/37] qapi/commands.py: enable checking with mypy

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/mypy.ini | 5 -
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index b668776d94..9da1dccef4 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.doc]
 disallow_subclassing_any = False
 disallow_untyped_defs = False
-- 
2.26.2

[PATCH 37/37] qapi/visit.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 72 +--
 2 files changed, 55 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 3babc380a8..5ab3433c5f 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ce71659a03..035521476f 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
 c_enum_const,
 c_name,
@@ -22,10 +24,19 @@
 INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaObjectTypeMember,
+QAPISchemaVariants,
+QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:
 c_type = c_name(name) + ' *'
 if not scalar:
 c_type += '*'
@@ -36,7 +47,7 @@ def gen_visit_decl(name, scalar=False):
  c_name=c_name(name), c_type=c_type)
 
 
-def gen_visit_members_decl(name):
+def gen_visit_members_decl(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp);
@@ -44,7 +55,10 @@ def gen_visit_members_decl(name):
  c_name=c_name(name))
 
 
-def gen_visit_object_members(name, base, members, variants):
+def gen_visit_object_members(name: str,
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> str:
 ret = mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -122,7 +136,7 @@ def gen_visit_object_members(name, base, members, variants):
 return ret
 
 
-def gen_visit_list(name, element_type):
+def gen_visit_list(name: str, element_type: QAPISchemaType) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -156,7 +170,7 @@ def gen_visit_list(name, element_type):
  c_name=c_name(name), c_elt_type=element_type.c_name())
 
 
-def gen_visit_enum(name):
+def gen_visit_enum(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj,
@@ -171,7 +185,7 @@ def gen_visit_enum(name):
  c_name=c_name(name))
 
 
-def gen_visit_alternate(name, variants):
+def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
 ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -247,7 +261,7 @@ def gen_visit_alternate(name, variants):
 return ret
 
 
-def gen_visit_object(name):
+def gen_visit_object(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -282,12 +296,12 @@ def gen_visit_object(name):
 
 class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
 ' * Built-in QAPI visitors', __doc__)
 
-def _begin_system_module(self, name):
+def _begin_system_module(self, name: None) -> None:
 self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/error.h"
@@ -299,7 +313,7 @@ def _begin_system_module(self, name):
 
 '''))
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 types = self._module_basename('qapi-types', name)
 visit = self._module_basename('qapi-visit', name)
 self._genc.preamble_add(mcgen('''
@@ -316,18 +330,34 @@ def _begin_user_module(self, name):
 ''',
   types=types))
 
-def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+def visit_enum_type(self,
+name: str,
+info: QAPISourceInfo,
+ifcond: List[str],
+features: List[QAPISchemaFeature],
+members: List[QAPISchemaEnumMember],
+prefix: Optional[str]) -> None:
 with ifcontext(ifcond, self._genh, self._genc):
 self._genh.add(gen_visit_decl(name, scalar=True))

[PATCH 22/37] qapi/source.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/mypy.ini  |  5 -
 scripts/qapi/source.py | 31 ++-
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 9da1dccef4..43c8bd1973 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -39,11 +39,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index e97b9a8e15..1cc6a5b82d 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-def __init__(self):
+def __init__(self) -> None:
 # Are documentation comments required?
 self.doc_required = False
 # Whitelist of commands allowed to return a non-dictionary
-self.returns_whitelist = []
+self.returns_whitelist: List[str] = []
 # Whitelist of entities allowed to violate case conventions
-self.name_case_whitelist = []
+self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-def __init__(self, fname, line, parent):
+T = TypeVar('T', bound='QAPISourceInfo')
+
+def __init__(self: T, fname: str, line: int, parent: Optional[T]):
 self.fname = fname
 self.line = line
 self.parent = parent
-self.pragma = parent.pragma if parent else QAPISchemaPragma()
-self.defn_meta = None
-self.defn_name = None
+self.pragma: QAPISchemaPragma = (
+parent.pragma if parent else QAPISchemaPragma()
+)
+self.defn_meta: Optional[str] = None
+self.defn_name: Optional[str] = None
 
-def set_defn(self, meta, name):
+def set_defn(self, meta: str, name: str) -> None:
 self.defn_meta = meta
 self.defn_name = name
 
-def next_line(self):
+def next_line(self: T) -> T:
 info = copy.copy(self)
 info.line += 1
 return info
 
-def loc(self):
+def loc(self) -> str:
 if self.fname is None:
 return sys.argv[0]
 ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
 ret += ':%d' % self.line
 return ret
 
-def in_defn(self):
+def in_defn(self) -> str:
 if self.defn_name:
 return "%s: In %s '%s':\n" % (self.fname,
   self.defn_meta, self.defn_name)
 return ''
 
-def include_path(self):
+def include_path(self) -> str:
 ret = ''
 parent = self.parent
 while parent:
@@ -63,5 +68,5 @@ def include_path(self):
 parent = parent.parent
 return ret
 
-def __str__(self):
+def __str__(self) -> str:
 return self.include_path() + self.in_defn() + self.loc()
-- 
2.26.2

[PATCH 24/37] qapi/gen.py: Fix edge-case of _is_user_module

2020-09-15 Thread John Snow

The edge case is that if the name is '', this expression returns a
string instead of a bool, which violates our declared type.

Signed-off-by: John Snow 
---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 11472ba043..fc46813d1a 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -221,7 +221,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, 
pydoc):
 
 @staticmethod
 def _is_user_module(name):
-return name and not name.startswith('./')
+return name is not None and not name.startswith('./')
 
 @staticmethod
 def _is_builtin_module(name):
-- 
2.26.2

[PATCH 19/37] qapi/commands.py: Don't re-bind to variable of different type

2020-09-15 Thread John Snow

Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.

Signed-off-by: John Snow 
---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 0c0fe854fe..b215e58357 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -194,14 +194,12 @@ def gen_register_command(name, success_response, 
allow_oob, allow_preconfig):
 if not options:
 options = ['QCO_NO_OPTIONS']
 
-options = " | ".join(options)
-
 ret = mcgen('''
 qmp_register_command(cmds, "%(name)s",
  qmp_marshal_%(c_name)s, %(opts)s);
 ''',
 name=name, c_name=c_name(name),
-opts=options)
+opts=" | ".join(options))
 return ret
 
 
-- 
2.26.2

[PATCH 34/37] qapi/types.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ---
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 9ce8b56f22..3babc380a8 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 53b47f9e58..274d62fac8 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,6 +13,8 @@
 # See the COPYING file in the top-level directory.
 """
 
+from typing import Optional, List
+
 from .common import (
 c_enum_const,
 c_name,
@@ -21,7 +23,16 @@
 mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .source import QAPISourceInfo
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaObjectType,
+QAPISchemaVariants,
+QAPISchemaObjectTypeMember,
+QAPISchemaFeature,
+QAPISchemaType,
+)
 
 
 # variants must be emitted before their container; track what has already
@@ -29,7 +40,9 @@
 objects_seen = set()
 
 
-def gen_enum_lookup(name, members, prefix=None):
+def gen_enum_lookup(name: str,
+members: List[QAPISchemaEnumMember],
+prefix: Optional[str] = None) -> str:
 ret = mcgen('''
 
 const QEnumLookup %(c_name)s_lookup = {
@@ -54,7 +67,9 @@ def gen_enum_lookup(name, members, prefix=None):
 return ret
 
 
-def gen_enum(name, members, prefix=None):
+def gen_enum(name: str,
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str] = None) -> str:
 # append automatically generated _MAX value
 enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
 
@@ -88,7 +103,7 @@ def gen_enum(name, members, prefix=None):
 return ret
 
 
-def gen_fwd_object_or_array(name):
+def gen_fwd_object_or_array(name: str) -> str:
 return mcgen('''
 
 typedef struct %(c_name)s %(c_name)s;
@@ -96,7 +111,7 @@ def gen_fwd_object_or_array(name):
  c_name=c_name(name))
 
 
-def gen_array(name, element_type):
+def gen_array(name: str, element_type: QAPISchemaType) -> str:
 return mcgen('''
 
 struct %(c_name)s {
@@ -107,7 +122,7 @@ def gen_array(name, element_type):
  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_members(members):
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
 ret = ''
 for memb in members:
 ret += gen_if(memb.ifcond)
@@ -124,7 +139,10 @@ def gen_struct_members(members):
 return ret
 
 
-def gen_object(name, ifcond, base, members, variants):
+def gen_object(name: str, ifcond: List[str],
+   base: Optional[QAPISchemaObjectType],
+   members: List[QAPISchemaObjectTypeMember],
+   variants: Optional[QAPISchemaVariants]) -> str:
 if name in objects_seen:
 return ''
 objects_seen.add(name)
@@ -178,7 +196,7 @@ def gen_object(name, ifcond, base, members, variants):
 return ret
 
 
-def gen_upcast(name, base):
+def gen_upcast(name: str, base: QAPISchemaObjectType) -> str:
 # C makes const-correctness ugly.  We have to cast away const to let
 # this function work for both const and non-const obj.
 return mcgen('''
@@ -191,7 +209,7 @@ def gen_upcast(name, base):
  c_name=c_name(name), base=base.c_name())
 
 
-def gen_variants(variants):
+def gen_variants(variants: QAPISchemaVariants) -> str:
 ret = mcgen('''
 union { /* union tag is @%(c_name)s */
 ''',
@@ -215,7 +233,7 @@ def gen_variants(variants):
 return ret
 
 
-def gen_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name: str) -> str:
 ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj);
@@ -225,7 +243,7 @@ def gen_type_cleanup_decl(name):
 return ret
 
 
-def gen_type_cleanup(name):
+def gen_type_cleanup(name: str) -> str:
 ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj)
@@ -247,12 +265,12 @@ def gen_type_cleanup(name):
 
 class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-types', ' * Schema-defined QAPI types',
 ' * Built-in QAPI types', __doc__)
 
-def _begin_system_module(self, name):
+def _begin_system_module(self, name: None) -> None:
 self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
@@ -263,7 +281,7 @@ def _begin_system_module(self, name):
 #include "qapi/util.h"
 '''))
 
-

[PATCH 18/37] qapi/events.py: Move comments into docstrings

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/events.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 449a700ffe..0b49cf5fea 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -49,8 +49,10 @@ def gen_event_send_decl(name: str,
  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+"""
+Declare and initialize a qapi object, using parameters from `build_params`.
+"""
 assert not typ.variants
 ret = mcgen('''
 %(c_name)s param = {
-- 
2.26.2

[PATCH 12/37] qapi/common.py: check with pylint

2020-09-15 Thread John Snow

Remove qapi/common.py from the pylintrc ignore list.

Signed-off-by: John Snow 
---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c2bbb8e8e1..c800055589 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the blacklist. The
 # regex matches against base names, not paths.
-ignore-patterns=common.py,
-doc.py,
+ignore-patterns=doc.py,
 error.py,
 expr.py,
 gen.py,
-- 
2.26.2

[PATCH 16/37] qapi: establish mypy type-checking baseline

2020-09-15 Thread John Snow

Fix two very minor issues, and then establish a mypy type-checking
baseline.

Like pylint, this should be run from the folder above:

 > mypy --config-file=qapi/mypy.ini qapi/

Signed-off-by: John Snow 
---
 scripts/qapi/doc.py|  3 +-
 scripts/qapi/mypy.ini  | 65 ++
 scripts/qapi/schema.py |  3 +-
 3 files changed, 69 insertions(+), 2 deletions(-)
 create mode 100644 scripts/qapi/mypy.ini

diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
index cbf7076ed9..70f7cdfaa6 100644
--- a/scripts/qapi/doc.py
+++ b/scripts/qapi/doc.py
@@ -5,7 +5,8 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor
 
 
 MSG_FMT = """
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
new file mode 100644
index 00..a0f2365a53
--- /dev/null
+++ b/scripts/qapi/mypy.ini
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index b4921b46c9..bb0cd717f1 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -17,6 +17,7 @@
 import os
 import re
 from collections import OrderedDict
+from typing import Optional
 
 from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-meta = None
+meta: Optional[str] = None
 
 def __init__(self, name, info, doc, ifcond=None, features=None):
 assert name is None or isinstance(name, str)
-- 
2.26.2

[PATCH 29/37] qapi/gen.py: delint with pylint

2020-09-15 Thread John Snow

'fp' and 'fd' are self-evident in context, add them to the list of OK
names.

_top and _bottom also need to stay class methods because some users
override the method and need to use `self`. Tell pylint to shush.

Signed-off-by: John Snow 
---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index bd67f6b069..0a72aecdd0 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -46,9 +46,11 @@ def get_content(self) -> str:
 return self._top() + self._preamble + self._body + self._bottom()
 
 def _top(self) -> str:
+# pylint: disable=no-self-use
 return ''
 
 def _bottom(self) -> str:
+# pylint: disable=no-self-use
 return ''
 
 def write(self, output_dir: str) -> None:
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c50a6656b7..3aebb0d228 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -5,7 +5,6 @@
 ignore-patterns=doc.py,
 error.py,
 expr.py,
-gen.py,
 parser.py,
 schema.py,
 types.py,
@@ -46,7 +45,9 @@ good-names=i,
k,
ex,
Run,
-   _
+   _,
+   fp,  # fp = open(...)
+   fd,  # fd = os.open(...)
 
 [VARIABLES]
 
-- 
2.26.2

[PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index c665e67495..4c079755d3 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -30,14 +30,14 @@ def camel_to_upper(value):
 new_name = ''
 length = len(c_fun_str)
 for i in range(length):
-c = c_fun_str[i]
-# When c is upper and no '_' appears before, do more checks
-if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+char = c_fun_str[i]
+# When char is upper and no '_' appears before, do more checks
+if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
 if i < length - 1 and c_fun_str[i + 1].islower():
 new_name += '_'
 elif c_fun_str[i - 1].isdigit():
 new_name += '_'
-new_name += c
+new_name += char
 return new_name.lstrip('_').upper()
 
 
-- 
2.26.2

[PATCH 15/37] qapi/common.py: split build_params into new file

2020-09-15 Thread John Snow

Including it in common.py creates a circular import dependency, because
schema relies on common.py. To type build_params properly, it needs to
be moved outside of the chain.

Signed-off-by: John Snow 
---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 23 ---
 scripts/qapi/events.py   |  2 +-
 scripts/qapi/params.py   | 40 
 4 files changed, 42 insertions(+), 25 deletions(-)
 create mode 100644 scripts/qapi/params.py

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 2e4b4de0fa..0c0fe854fe 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -14,10 +14,10 @@
 """
 
 from .common import (
-build_params,
 c_name,
 mcgen,
 )
+from .params import build_params
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 38d380f0a9..0b1af694e6 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
 return ret
-
-
-def build_params(arg_type,
- boxed: bool,
- extra: Optional[str] = None) -> str:
-ret = ''
-sep = ''
-if boxed:
-assert arg_type
-ret += '%s arg' % arg_type.c_param_type()
-sep = ', '
-elif arg_type:
-assert not arg_type.variants
-for memb in arg_type.members:
-ret += sep
-sep = ', '
-if memb.optional:
-ret += 'bool has_%s, ' % c_name(memb.name)
-ret += '%s %s' % (memb.type.c_param_type(),
-  c_name(memb.name))
-if extra:
-ret += sep + extra
-return ret if ret else 'void'
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 6b3afa14d7..75eda72534 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -13,11 +13,11 @@
 """
 
 from .common import (
-build_params,
 c_enum_const,
 c_name,
 mcgen,
 )
+from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/params.py b/scripts/qapi/params.py
new file mode 100644
index 00..4d4b02f60d
--- /dev/null
+++ b/scripts/qapi/params.py
@@ -0,0 +1,40 @@
+#
+# QAPI helper library
+#
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2018 Red Hat Inc.
+#
+# Authors:
+#  Anthony Liguori 
+#  Markus Armbruster 
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from typing import Optional
+
+from .common import c_name
+from .schema import QAPISchemaObjectType
+
+
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
+ret = ''
+sep = ''
+if boxed:
+assert arg_type
+ret += '%s arg' % arg_type.c_param_type()
+sep = ', '
+elif arg_type:
+assert not arg_type.variants
+for memb in arg_type.members:
+ret += sep
+sep = ', '
+if memb.optional:
+ret += 'bool has_%s, ' % c_name(memb.name)
+ret += '%s %s' % (memb.type.c_param_type(),
+  c_name(memb.name))
+if extra:
+ret += sep + extra
+return ret if ret else 'void'
-- 
2.26.2

[PATCH 23/37] qapi/source.py: delint with pylint

2020-09-15 Thread John Snow

Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.

Signed-off-by: John Snow 
---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c800055589..c50a6656b7 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -8,7 +8,6 @@ ignore-patterns=doc.py,
 gen.py,
 parser.py,
 schema.py,
-source.py,
 types.py,
 visit.py,
 
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 1cc6a5b82d..ba991d798f 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -15,6 +15,9 @@
 
 
 class QAPISchemaPragma:
+# Replace with @dataclass in Python 3.7+
+# pylint: disable=too-few-public-methods
+
 def __init__(self) -> None:
 # Are documentation comments required?
 self.doc_required = False
-- 
2.26.2

[PATCH 09/37] qapi/common.py: Add indent manager

2020-09-15 Thread John Snow

Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.

Make a little indent level manager instead.

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 50 +-
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 38 insertions(+), 19 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 4fb265a8bf..87d87b95e5 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-ret = ''
-for _ in range(count):
-ret += ' '
-return ret
+class Indent:
+"""
+Indent-level management.
 
+:param initial: Initial value, default 0.
+"""
+def __init__(self, initial: int = 0) -> None:
+self._level = initial
 
-indent_level = 0
+def __int__(self) -> int:
+"""Return the indent as an integer."""
+return self._level
 
+def __repr__(self) -> str:
+return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-global indent_level
-indent_level += indent_amount
+def __str__(self) -> str:
+"""Return the indent as a string."""
+return ' ' * self._level
 
+def __bool__(self) -> bool:
+"""True when there is a non-zero indent."""
+return bool(self._level)
 
-def pop_indent(indent_amount=4):
-global indent_level
-indent_level -= indent_amount
+def push(self, amount: int = 4) -> int:
+"""Push `amount` spaces onto the indent, default four."""
+self._level += amount
+return self._level
+
+def pop(self, amount: int = 4) -> int:
+"""Pop `amount` spaces off of the indent, default four."""
+if self._level < amount:
+raise ArithmeticError(
+"Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
+self._level -= amount
+return self._level
+
+
+INDENT = Indent(0)
 
 
 # Generate @code with @kwds interpolated.
 # Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
 raw = code % kwds
-if indent_level:
-indent = genindent(indent_level)
-raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+if INDENT:
+raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
 return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 31bf46e7f7..66ce3dc52e 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -19,8 +19,7 @@
 gen_endif,
 gen_if,
 mcgen,
-pop_indent,
-push_indent,
+INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
 if (visit_optional(v, "%(name)s", >has_%(c_name)s)) {
 ''',
  name=memb.name, c_name=c_name(memb.name))
-push_indent()
+INDENT.push()
 ret += mcgen('''
 if (!visit_type_%(c_type)s(v, "%(name)s", >%(c_name)s, errp)) {
 return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
  c_type=memb.type.c_name(), name=memb.name,
  c_name=c_name(memb.name))
 if memb.optional:
-pop_indent()
+INDENT.pop()
 ret += mcgen('''
 }
 ''')
-- 
2.26.2

[PATCH 08/37] qapi/common.py: Remove python compatibility workaround

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index ba35abea47..4fb265a8bf 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
 raw = code % kwds
 if indent_level:
 indent = genindent(indent_level)
-# re.subn() lacks flags support before Python 2.7, use re.compile()
-raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-  indent, raw)
-raw = raw[0]
+raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
 return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
-- 
2.26.2

[PATCH 13/37] qapi/common.py: add notational type hints

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 27 ---
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 4c079755d3..af01348b35 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Union, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
 c_fun_str = c_name(value, False)
 if value.isupper():
 return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
 return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+ const_name: str,
+ prefix: Optional[str] = None) -> str:
 if prefix is not None:
 type_name = prefix
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
 # ANSI X3J11/88-090, 3.1.1
 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
  'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
 
 # Generate @code with @kwds interpolated.
 # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:
 raw = code % kwds
 if INDENT:
 raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
 return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:
 if code[0] == '\n':
 code = code[1:]
 return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
 return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
 return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -160,7 +163,7 @@ def guardstart(name):
  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
 return mcgen('''
 
 #endif /* %(name)s */
@@ -168,7 +171,7 @@ def guardend(name):
  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
 ret = ''
 for ifc in ifcond:
 ret += mcgen('''
@@ -177,7 +180,7 @@ def gen_if(ifcond):
 return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
 ret = ''
 for ifc in reversed(ifcond):
 ret += mcgen('''
@@ -186,7 +189,9 @@ def gen_endif(ifcond):
 return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
 ret = ''
 sep = ''
 if boxed:
-- 
2.26.2

[PATCH 14/37] qapi/common.py: Move comments into docstrings

2020-09-15 Thread John Snow

As docstrings, they'll show up in documentation and IDE help.

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 50 ++
 1 file changed, 36 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index af01348b35..38d380f0a9 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -20,10 +20,18 @@
 c_name_trans = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+"""
+Converts CamelCase to CAMEL_CASE.
+
+Examples:
+  ENUMName -> ENUM_NAME
+  EnumName1 -> ENUM_NAME1
+  ENUM_NAME -> ENUM_NAME
+  ENUM_NAME1 -> ENUM_NAME1
+  ENUM_Name2 -> ENUM_NAME2
+  ENUM24_Name -> ENUM24_NAME
+"""
 c_fun_str = c_name(value, False)
 if value.isupper():
 return c_fun_str
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
  const_name: str,
  prefix: Optional[str] = None) -> str:
+"""
+Generate a C enumeration constant name.
+
+:param type_name: The name of the enumeration.
+:param const_name: The name of this constant.
+:param prefix: Optional, prefix that overrides the type_name.
+"""
 if prefix is not None:
 type_name = prefix
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+"""
+Map `name` to a valid C identifier.
+
+Used for converting 'name' from a 'name':'type' qapi definition
+into a generated struct member, as well as converting type names
+into substrings of a generated C function name.
+
+'__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+:param name: The name to map.
+:param protect: If true, avoid returning certain ticklish identifiers
+(like C keywords) by prepending 'q_'.
+"""
 # ANSI X3J11/88-090, 3.1.1
 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
  'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
 INDENT = Indent(0)
 
 
-# Generate @code with @kwds interpolated.
-# Obey INDENT level, and strip EATSPACE.
 def cgen(code: str, **kwds: Union[str, int]) -> str:
+"""
+Generate `code` with `kwds` interpolated.
+Obey `INDENT`, and strip `EATSPACE`.
+"""
 raw = code % kwds
 if INDENT:
 raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-- 
2.26.2

[PATCH 05/37] qapi: Remove wildcard includes

2020-09-15 Thread John Snow

Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.

Remove them and include things explicitly by name instead.

Signed-off-by: John Snow 
---
 scripts/qapi/commands.py   |  6 +-
 scripts/qapi/events.py |  7 ++-
 scripts/qapi/gen.py| 12 +---
 scripts/qapi/introspect.py |  7 ++-
 scripts/qapi/types.py  |  8 +++-
 scripts/qapi/visit.py  | 10 +-
 6 files changed, 42 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index ce5926146a..e1df0e341f 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,11 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+build_params,
+c_name,
+mcgen,
+)
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 0467272438..6b3afa14d7 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,7 +12,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+build_params,
+c_enum_const,
+c_name,
+mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 8df19a0df0..11472ba043 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+c_fname,
+gen_endif,
+gen_if,
+guardend,
+guardstart,
+mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 2a34cd1e8e..b036fcf9ce 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
  QAPISchemaType)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index ca9a5aacb3..53b47f9e58 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,7 +13,13 @@
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_enum_const,
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 7850f6e848..ea277e7704 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,7 +13,15 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_enum_const,
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+pop_indent,
+push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
-- 
2.26.2

[PATCH 06/37] qapi: delint using flake8

2020-09-15 Thread John Snow

Petty style guide fixes and line length enforcement.  Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.

Signed-off-by: John Snow 
---
 scripts/qapi/.flake8 |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +---
 scripts/qapi/visit.py| 15 ++-
 4 files changed, 19 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8

diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
new file mode 100644
index 00..45d8146f3f
--- /dev/null
+++ b/scripts/qapi/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index e1df0e341f..2e4b4de0fa 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
 return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject 
**ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+  Error **errp)
 {
 Visitor *v;
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index a835ee6fde..b77e0e56b2 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
 v.set_defined_in(name)
 
 def check(self, schema, seen):
-if not self.tag_member: # flat union
+if not self.tag_member:  # flat union
 self.tag_member = seen.get(c_name(self._tag_name))
 base = "'base'"
 # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
 self._entity_dict = {}
 self._module_dict = OrderedDict()
 self._schema_dir = os.path.dirname(fname)
-self._make_module(None) # built-ins
+self._make_module(None)  # built-ins
 self._make_module(fname)
 self._predefining = True
 self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, 
role, members):
 # But it's not tight: the disjunction need not imply it.  We
 # may end up compiling useless wrapper types.
 # TODO kill simple unions or implement the disjunction
-assert (ifcond or []) == typ._ifcond # pylint: 
disable=protected-access
+
+# pylint: disable=protected-access
+assert (ifcond or []) == typ._ifcond
 else:
 self._def_entity(QAPISchemaObjectType(
 name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ea277e7704..31bf46e7f7 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -31,7 +31,8 @@ def gen_visit_decl(name, scalar=False):
 if not scalar:
 c_type += '*'
 return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error 
**errp);
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj,
+   Error **errp);
 ''',
  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +126,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+   Error **errp)
 {
 bool ok = false;
 %(c_name)s *tail;
@@ -158,7 +160,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj,
+   Error **errp)
 {
 int value = *obj;
 bool ok = visit_type_enum(v, name, , &%(c_name)s_lookup, errp);
@@ -172,7 +175,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
 ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+   Error **errp)
 {
 bool ok = false;
 
@@ -247,7 +251,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+   Error **errp)
 {
 bool ok = false;
 
-- 
2.26.2

[PATCH 07/37] qapi: add pylintrc

2020-09-15 Thread John Snow

Add a skeleton pylintrc file. Right now, it ignores quite a few things.
Files will be removed from the known-bad list throughout this and
following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder now.

Due to some bugs in different versions of pylint (2.5.x), pylint does
not correctly recognize when it is being run from "inside" a module, and
must be run *outside* of the module.

Therefore, to run it, you must:

 > cd :/qemu/scripts
 > pylint qapi/ --rcfile=qapi/pylintrc

Signed-off-by: John Snow 
---
 scripts/qapi/pylintrc | 74 +++
 1 file changed, 74 insertions(+)
 create mode 100644 scripts/qapi/pylintrc

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
new file mode 100644
index 00..c2bbb8e8e1
--- /dev/null
+++ b/scripts/qapi/pylintrc
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.
+ignore-patterns=common.py,
+doc.py,
+error.py,
+expr.py,
+gen.py,
+parser.py,
+schema.py,
+source.py,
+types.py,
+visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+missing-docstring,
+too-many-arguments,
+too-many-branches,
+too-many-statements,
+too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+   j,
+   k,
+   ex,
+   Run,
+   _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
-- 
2.26.2

[PATCH 03/37] qapi-gen: Separate arg-parsing from generation

2020-09-15 Thread John Snow

This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.

Signed-off-by: John Snow 
---
 scripts/qapi-gen.py | 87 -
 1 file changed, 63 insertions(+), 24 deletions(-)

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 4b03f7d53b..59becba3e1 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,9 +1,13 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
@@ -11,21 +15,65 @@
 
 from qapi.commands import gen_commands
 from qapi.doc import gen_doc
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+ output_dir: str,
+ prefix: str,
+ unmask: bool = False,
+ builtins: bool = False) -> None:
+"""
+generate uses a given schema to produce C code in the target directory.
+
+:param schema_file: The primary QAPI schema file.
+:param output_dir: The output directory to store generated code.
+:param prefix: Optional C-code prefix for symbol names.
+:param unmask: Expose non-ABI names through introspection?
+:param builtins: Generate code for built-in types?
+
+:raise QAPIError: On failures.
+"""
+match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+if match and match.end() != len(prefix):
+msg = "funny character '{:s}' in prefix '{:s}'".format(
+prefix[match.end()], prefix)
+raise QAPIError('', None, msg)
+
+schema = QAPISchema(schema_file)
+gen_types(schema, output_dir, prefix, builtins)
+gen_visit(schema, output_dir, prefix, builtins)
+gen_commands(schema, output_dir, prefix)
+gen_events(schema, output_dir, prefix)
+gen_introspect(schema, output_dir, prefix, unmask)
+gen_doc(schema, output_dir, prefix)
+
+
+def main() -> int:
+"""
+gapi-gen shell script entrypoint.
+Expects arguments via sys.argv, see --help for details.
+
+:return: int, 0 on success, 1 on failure.
+"""
 parser = argparse.ArgumentParser(
 description='Generate code from a QAPI schema')
 parser.add_argument('-b', '--builtins', action='store_true',
 help="generate code for built-in types")
-parser.add_argument('-o', '--output-dir', action='store', default='',
+parser.add_argument('-o', '--output-dir', action='store',
+default=DEFAULT_OUTPUT_DIR,
 help="write output to directory OUTPUT_DIR")
-parser.add_argument('-p', '--prefix', action='store', default='',
+parser.add_argument('-p', '--prefix', action='store',
+default=DEFAULT_PREFIX,
 help="prefix for symbols")
 parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
 dest='unmask',
@@ -33,26 +81,17 @@ def main(argv):
 parser.add_argument('schema', action='store')
 args = parser.parse_args()
 
-match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-if match.end() != len(args.prefix):
-print("%s: 'funny character '%s' in argument of --prefix"
-  % (sys.argv[0], args.prefix[match.end()]),
-  file=sys.stderr)
-sys.exit(1)
-
 try:
-schema = QAPISchema(args.schema)
+generate(args.schema,
+ output_dir=args.output_dir,
+ prefix=args.prefix,
+ unmask=args.unmask,
+ builtins=args.builtins)
 except QAPIError as err:
-print(err, file=sys.stderr)
-exit(1)
-
-gen_types(schema, args.output_dir, args.prefix, args.builtins)
-gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-gen_commands(schema, args.output_dir, args.prefix)
-gen_events(schema, args.output_dir, args.prefix)
-gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
-gen_doc(schema, args.output_dir, args.prefix)
+print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+return 1
+return 0
 
 
 if __name__ == '__main__':
-main(sys.argv)
+sys.exit(main())
-- 
2.26.2

[PATCH 10/37] qapi/common.py: delint with pylint

2020-09-15 Thread John Snow

Signed-off-by: John Snow 
---
 scripts/qapi/common.py | 16 +++-
 scripts/qapi/schema.py | 14 +++---
 2 files changed, 14 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 87d87b95e5..c665e67495 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
 return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indent:
 """
 Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.
 def cgen(code, **kwds):
 raw = code % kwds
 if INDENT:
 raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-return re.sub(re.escape(eatspace) + r' *', '', raw)
+return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index b77e0e56b2..b4921b46c9 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@
 import re
 from collections import OrderedDict
 
-from .common import c_name, pointer_suffix
+from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
 return True
 
 def c_type(self):
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def json_type(self):
 return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
 
 def c_type(self):
 assert not self.is_implicit()
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def c_unboxed_type(self):
 return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
 v.connect_doc(doc)
 
 def c_type(self):
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def json_type(self):
 return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
 self._make_array_type(name, None)
 
 def _def_predefineds(self):
-for t in [('str','string',  'char' + pointer_suffix),
+for t in [('str','string',  'char' + POINTER_SUFFIX),
   ('number', 'number',  'double'),
   ('int','int', 'int64_t'),
   ('int8',   'int', 'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
   ('uint64', 'int', 'uint64_t'),
   ('size',   'int', 'uint64_t'),
   ('bool',   'boolean', 'bool'),
-  ('any','value',   'QObject' + pointer_suffix),
-  ('null',   'null','QNull' + pointer_suffix)]:
+  ('any','value',   'QObject' + POINTER_SUFFIX),
+  ('null',   'null','QNull' + POINTER_SUFFIX)]:
 self._def_builtin_type(*t)
 self.the_empty_object_type = QAPISchemaObjectType(
 'q_empty', None, None, None, None, None, [], None)
-- 
2.26.2

[PATCH 04/37] qapi: move generator entrypoint into module

2020-09-15 Thread John Snow

As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the module, only leaving a very
tiny entrypoint shim outside of the module.

As a result, all of the include statements are reworked to be module-aware,
as explicit relative imports.

This is done primarily for the benefit of python tooling (pylint, mypy,
flake8, et al) which otherwise has trouble consistently resolving
"qapi.x" to mean "a sibling file in this folder."

Signed-off-by: John Snow 
---
 scripts/qapi-gen.py| 94 +++---
 scripts/qapi/commands.py   |  4 +-
 scripts/qapi/doc.py|  2 +-
 scripts/qapi/events.py |  8 ++--
 scripts/qapi/expr.py   |  4 +-
 scripts/qapi/gen.py|  4 +-
 scripts/qapi/introspect.py |  8 ++--
 scripts/qapi/parser.py |  4 +-
 scripts/qapi/schema.py |  8 ++--
 scripts/qapi/script.py | 91 
 scripts/qapi/types.py  |  6 +--
 scripts/qapi/visit.py  |  6 +--
 12 files changed, 124 insertions(+), 115 deletions(-)
 create mode 100644 scripts/qapi/script.py

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 59becba3e1..e649f8dd44 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,97 +1,15 @@
 #!/usr/bin/env python3
-
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This script is the main entry point for generating C code from the QAPI schema.
+This file exists primarily to facilitate the running of the QAPI code
+generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.doc import gen_doc
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
- output_dir: str,
- prefix: str,
- unmask: bool = False,
- builtins: bool = False) -> None:
-"""
-generate uses a given schema to produce C code in the target directory.
-
-:param schema_file: The primary QAPI schema file.
-:param output_dir: The output directory to store generated code.
-:param prefix: Optional C-code prefix for symbol names.
-:param unmask: Expose non-ABI names through introspection?
-:param builtins: Generate code for built-in types?
-
-:raise QAPIError: On failures.
-"""
-match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-if match and match.end() != len(prefix):
-msg = "funny character '{:s}' in prefix '{:s}'".format(
-prefix[match.end()], prefix)
-raise QAPIError('', None, msg)
-
-schema = QAPISchema(schema_file)
-gen_types(schema, output_dir, prefix, builtins)
-gen_visit(schema, output_dir, prefix, builtins)
-gen_commands(schema, output_dir, prefix)
-gen_events(schema, output_dir, prefix)
-gen_introspect(schema, output_dir, prefix, unmask)
-gen_doc(schema, output_dir, prefix)
-
-
-def main() -> int:
-"""
-gapi-gen shell script entrypoint.
-Expects arguments via sys.argv, see --help for details.
-
-:return: int, 0 on success, 1 on failure.
-"""
-parser = argparse.ArgumentParser(
-description='Generate code from a QAPI schema')
-parser.add_argument('-b', '--builtins', action='store_true',
-help="generate code for built-in types")
-parser.add_argument('-o', '--output-dir', action='store',
-default=DEFAULT_OUTPUT_DIR,
-help="write output to directory OUTPUT_DIR")
-parser.add_argument('-p', '--prefix', action='store',
-default=DEFAULT_PREFIX,
-help="prefix for symbols")
-parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-dest='unmask',
-help="expose non-ABI names in introspection")
-parser.add_argument('schema', action='store')
-args = parser.parse_args()
-
-try:
-generate(args.schema,
- output_dir=args.output_dir,
- prefix=args.prefix,
- unmask=args.unmask,
- builtins=args.builtins)
-except QAPIError as err:
-print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-return 1
-return 0
-
+from qapi import script
 
 if __name__ == '__main__':
-sys.exit(main())
+sys.exit(script.main())
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 3cf9e1110b..ce5926146a 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,8

[PATCH 00/37] qapi: static typing conversion, pt1

2020-09-15 Thread John Snow

Hi, this series starts adding static type hints to the QAPI module. As
you can see, the series started getting quite a bit long, so this is
only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive
their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures,
  which have no runtime impact whatsoever. These are big patches,
  but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one
  single warning.

- After patch 6, `flake8 qapi/` should pass 100% on this and every
  future commit.

- After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
  on this and every future commit.

- After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
  100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
003/37: 'qapi-gen: Separate arg-parsing from generation'
004/37: 'qapi: move generator entrypoint into module'
005/37: 'qapi: Remove wildcard includes'
006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround'
009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable'
012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints'
014/37: 'qapi/common.py: Move comments into docstrings'
015/37: 'qapi/common.py: split build_params into new file'
016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints'
018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
020/37: 'qapi/commands.py: add notational type hints'
021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints'
023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
025/37: 'qapi/gen.py: add notational type hints'
026/37: 'qapi/gen.py: Enable checking with mypy'
027/37: 'qapi/gen.py: Remove unused parameter'
028/37: 'qapi/gen.py: update write() to be more idiomatic'
029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
031/37: 'qapi/introspect.py: add _gen_features helper'
032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints'
035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
037/37: 'qapi/visit.py: add notational type hints'

John Snow (37):
  python: Require 3.6+
  [DO-NOT-MERGE] qapi: add debugging tools
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into module
  qapi: Remove wildcard includes
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add notational type hints
  qapi/common.py: Move comments into docstrings
  qapi/common.py: split build_params into new file
  qapi: establish mypy type-checking baseline
  qapi/events.py: add notational type hints
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add notational type hints
  qapi/commands.py: enable checking with mypy
  qapi/source.py: add notational type hints
  qapi/source.py: delint with pylint
  qapi/gen.py: Fix edge-case of _is_user_module
  qapi/gen.py: add notational type hints
  qapi/gen.py: Enable checking with mypy
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/introspect.py: Add a typed 'extra' structure
  qapi/introspect.py: add _gen_features helper
  qapi/introspect.py: create a typed 'Node' data structure
  qapi/introspect.py: add notational type hints
  qapi/types.py: add notational type hints
  qapi/types.py: remove one-letter variables
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add notational type hints

 docs/conf.py  |   4 +-
 configure |   6 +-
 .readthedocs.yml  |   2 +-
 .travis.yml   |   8 --
 scripts/qapi-gen.py   |  59 ++
 scripts/qapi/.flake8  |   2 +
 scripts/qapi/commands.py  |

[PATCH 02/37] [DO-NOT-MERGE] qapi: add debugging tools

2020-09-15 Thread John Snow

This adds some really childishly simple debugging tools. Maybe they're
interesting for someone else, too?

Signed-off-by: John Snow 
---
 scripts/qapi/debug.py | 78 +++
 1 file changed, 78 insertions(+)
 create mode 100644 scripts/qapi/debug.py

diff --git a/scripts/qapi/debug.py b/scripts/qapi/debug.py
new file mode 100644
index 00..bacf5ee180
--- /dev/null
+++ b/scripts/qapi/debug.py
@@ -0,0 +1,78 @@
+"""
+Small debugging facilities for mypy static analysis work.
+(C) 2020 John Snow, for Red Hat, Inc.
+"""
+
+import inspect
+import json
+from typing import Dict, List, Any
+from types import FrameType
+
+
+OBSERVED_TYPES: Dict[str, List[str]] = {}
+
+
+# You have no idea how long it took to find this return type...
+def caller_frame() -> FrameType:
+"""
+Returns the stack frame of the caller's caller.
+e.g. foo() -> caller() -> caller_frame() return's foo's stack frame.
+"""
+stack = inspect.stack()
+caller = stack[2].frame
+if caller is None:
+msg = "Python interpreter does not support stack frame inspection"
+raise RuntimeError(msg)
+return caller
+
+
+def _add_type_record(name: str, typestr: str) -> None:
+seen = OBSERVED_TYPES.setdefault(name, [])
+if typestr not in seen:
+seen.append(typestr)
+
+
+def record_type(name: str, value: Any, dict_names: bool = False) -> None:
+"""
+Record the type of a variable.
+
+:param name: The name of the variable
+:param value: The value of the variable
+"""
+_add_type_record(name, str(type(value)))
+
+try:
+for key, subvalue in value.items():
+subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]"
+_add_type_record(subname, str(type(subvalue)))
+return
+except AttributeError:
+# (Wasn't a dict or anything resembling one.)
+pass
+
+# str is iterable, but not in the way we want!
+if isinstance(value, str):
+return
+
+try:
+for elem in value:
+_add_type_record(f"{name}.[list_elem]", str(type(elem)))
+except TypeError:
+# (Wasn't a list or anything else iterable.)
+pass
+
+
+def show_types() -> None:
+"""
+Print all of the currently known variable types to stdout.
+"""
+print(json.dumps(OBSERVED_TYPES, indent=2))
+
+
+def record_locals(show: bool = False, dict_names: bool = False) -> None:
+caller = caller_frame()
+name = caller.f_code.co_name
+for key, value in caller.f_locals.items():
+record_type(f"{name}.{key}", value, dict_names=dict_names)
+if show:
+show_types()
-- 
2.26.2

[PATCH 01/37] python: Require 3.6+

2020-09-15 Thread John Snow

Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *

PEP 526 in particular will allow us to convert the QAPI module to the
statically typed subset of the Python language. These static type hints
improve the interactive editing experience in IDEs and help make
refactoring and adding new features faster, easier and safer.

*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict
itself is still allowed to have an arbitrary order based on
implementation. It is not until Python 3.7 that the dict type guarantees
insertion order.

Signed-off-by: John Snow 
---
 docs/conf.py  | 4 ++--
 configure | 6 +++---
 .readthedocs.yml  | 2 +-
 .travis.yml   | 8 
 tests/qemu-iotests/iotests.py | 2 --
 5 files changed, 6 insertions(+), 16 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 0dbd90dc11..8aeac40124 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -36,9 +36,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,5):
+if sys.version_info < (3,6):
 raise ConfigError(
-"QEMU requires a Sphinx that uses Python 3.5 or better\n")
+"QEMU requires a Sphinx that uses Python 3.6 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
diff --git a/configure b/configure
index ce27eafb0a..33292500e7 100755
--- a/configure
+++ b/configure
@@ -913,7 +913,7 @@ fi
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
+# We prefer python 3.6+. A bare 'python' is traditionally
 # python 2.x, but some distros have it as python 3.x, so
 # we check that too
 python=
@@ -1961,8 +1961,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
-  error_exit "Cannot use '$python', Python >= 3.5 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+  error_exit "Cannot use '$python', Python >= 3.6 is required." \
   "Use --python=/path/to/python to specify a supported Python."
 fi
 
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 8355dbc634..7fb7b8dd61 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -17,4 +17,4 @@ formats: all
 # we require for other Python in our codebase (our conf.py
 # enforces this, and some code needs it.)
 python:
-  version: 3.5
+  version: 3.6
diff --git a/.travis.yml b/.travis.yml
index 65341634d0..0dca15a5e8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -267,14 +267,6 @@ jobs:
 
 
 # Python builds
-- name: "GCC Python 3.5 (x86_64-softmmu)"
-  env:
-- CONFIG="--target-list=x86_64-softmmu"
-- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-  language: python
-  python: 3.5
-
-
 - name: "GCC Python 3.6 (x86_64-softmmu)"
   env:
 - CONFIG="--target-list=x86_64-softmmu"
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 91e4a57126..f48460480a 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -40,8 +40,6 @@
 from qemu import qtest
 from qemu.qmp import QMPMessage
 
-assert sys.version_info >= (3, 6)
-
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
 logger.addHandler(logging.NullHandler())
-- 
2.26.2

Re: [PATCH 1/5] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Eduardo Habkost

On Tue, Sep 15, 2020 at 02:30:34PM -0700, Richard Henderson wrote:
> On 9/15/20 2:27 PM, Eduardo Habkost wrote:
> > On Tue, Sep 15, 2020 at 01:51:48PM -0700, Richard Henderson wrote:
> >> On 9/15/20 1:19 PM, Eduardo Habkost wrote:
> >>> Once we move to C11, we can just use max_align_t.
> >>
> >> Yes.
> >>
> >>> While we don't move to C11, why not just use
> >>>   __alignof__(union { long l; void *p; double d; long double ld;})
> >>> ?
> >>
> >> For i386, this is 4.
> > 
> > Is i386-linux the only case where there are additional alignment
> > guarantees not covered by C99?
> 
> I think so.
> 
> > I would prefer a i386-linux-specific #ifdef for that case instead
> > of guessing based on undocumented libc internals.
> 
> I was thinking abi, not internals.

I see.  As long as we can point to the specification backing the
assumptions in the code, I'm OK with that.

To me, it seems simpler to start with something that works on all
hosts (the alignment guarantees provided by C99), and add
arch-specific optimizations later.

-- 
Eduardo

[PATCH v3 0/5] Qemu SEV-ES guest support

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

This patch series provides support for launching an SEV-ES guest.

Secure Encrypted Virtualization - Encrypted State (SEV-ES) expands on the
SEV support to protect the guest register state from the hypervisor. See
"AMD64 Architecture Programmer's Manual Volume 2: System Programming",
section "15.35 Encrypted State (SEV-ES)" [1].

In order to allow a hypervisor to perform functions on behalf of a guest,
there is architectural support for notifying a guest's operating system
when certain types of VMEXITs are about to occur. This allows the guest to
selectively share information with the hypervisor to satisfy the requested
function. The notification is performed using a new exception, the VMM
Communication exception (#VC). The information is shared through the
Guest-Hypervisor Communication Block (GHCB) using the VMGEXIT instruction.
The GHCB format and the protocol for using it is documented in "SEV-ES
Guest-Hypervisor Communication Block Standardization" [2].

The main areas of the Qemu code that are updated to support SEV-ES are
around the SEV guest launch process and AP booting in order to support
booting multiple vCPUs.

There are no new command line switches required. Instead, the desire for
SEV-ES is presented using the SEV policy object. Bit 2 of the SEV policy
object indicates that SEV-ES is required.

The SEV launch process is updated in two ways. The first is that a the
KVM_SEV_ES_INIT ioctl is used to initialize the guest instead of the
standard KVM_SEV_INIT ioctl. The second is that before the SEV launch
measurement is calculated, the LAUNCH_UPDATE_VMSA SEV API is invoked for
each vCPU that Qemu has created. Once the LAUNCH_UPDATE_VMSA API has been
invoked, no direct changes to the guest register state can be made.

AP booting poses some interesting challenges. The INIT-SIPI-SIPI sequence
is typically used to boot the APs. However, the hypervisor is not allowed
to update the guest registers. For the APs, the reset vector must be known
in advance. An OVMF method to provide a known reset vector address exists
by providing an SEV information block, identified by UUID, near the end of
the firmware [3]. OVMF will program the jump to the actual reset vector in
this area of memory. Since the memory location is known in advance, an AP
can be created with the known reset vector address as its starting CS:IP.
The GHCB document [2] talks about how SMP booting under SEV-ES is
performed. SEV-ES also requires the use of the in-kernel irqchip support
in order to minimize the changes required to Qemu to support AP booting.

[1] https://www.amd.com/system/files/TechDocs/24593.pdf
[2] https://developer.amd.com/wp-content/resources/56421.pdf
[3] 30937f2f98c4 ("OvmfPkg: Use the SEV-ES work area for the SEV-ES AP reset 
vector")

https://github.com/tianocore/edk2/commit/30937f2f98c42496f2f143fe8374ae7f7e684847

---

These patches are based on commit:
d0ed6a69d3 ("Update version for v5.1.0 release")

(I tried basing on the latest Qemu commit, but I was having build issues
that level)

A version of the tree can be found at:
https://github.com/AMDESE/qemu/tree/sev-es-v11

Changes since v2:
- Add in-kernel irqchip requirement for SEV-ES guests

Changes since v1:
- Fixed checkpatch.pl errors/warnings

Tom Lendacky (5):
  sev/i386: Add initial support for SEV-ES
  sev/i386: Require in-kernel irqchip support for SEV-ES guests
  sev/i386: Allow AP booting under SEV-ES
  sev/i386: Don't allow a system reset under an SEV-ES guest
  sev/i386: Enable an SEV-ES guest based on SEV policy

 accel/kvm/kvm-all.c   |  73 ++
 accel/stubs/kvm-stub.c|   5 ++
 hw/i386/pc_sysfw.c|  10 +++-
 include/sysemu/cpus.h |   2 +
 include/sysemu/hw_accel.h |   5 ++
 include/sysemu/kvm.h  |  18 +++
 include/sysemu/sev.h  |   3 ++
 softmmu/cpus.c|   5 ++
 softmmu/vl.c  |   5 +-
 target/i386/cpu.c |   1 +
 target/i386/kvm.c |   2 +
 target/i386/sev-stub.c|   5 ++
 target/i386/sev.c | 105 +-
 target/i386/sev_i386.h|   1 +
 14 files changed, 236 insertions(+), 4 deletions(-)

-- 
2.28.0

[ANNOUNCE] QEMU 5.0.1 Stable released

2020-09-15 Thread Michael Roth

Hi everyone,

I am pleased to announce that the QEMU v5.0.1 stable release is now
available:

You can grab the tarball from our download page here:

  https://www.qemu.org/download/#source

v5.0.1 is now tagged in the official qemu.git repository,
and the stable-5.0 branch has been updated accordingly:

  https://git.qemu.org/?p=qemu.git;a=shortlog;h=refs/heads/stable-5.0

This update contains general fixes for various architectures/subsystems,
including the following CVE fixes:

ati-vga (CVE-2020-13800)
MMIO (CVE-2020-13754)
NBD server (CVE-2020-10761)
sdcard (CVE-2020-13253)
usb (CVE-2020-14364)
virtiofsd (CVE-2020-10717)

Please see the changelog for additional details and update accordingly.

Thank you to everyone involved!

CHANGELOG:

386b2a5767: Update version for 5.0.1 release (Michael Roth)
5c49f7ee3b: riscv: sifive_test: Allow 16-bit writes to memory region (Nathan 
Chancellor)
b8fdfa9d63: virtio-ccw: fix virtio_set_ind_atomic (Halil Pasic)
ebf5b3946e: nvram: Exit QEMU if NVRAM cannot contain all -prom-env data (Greg 
Kurz)
f2fd6555c7: 9p: null terminate fs driver options list (Prasad J Pandit)
f243bb4b0d: usb: fix setup_len init (CVE-2020-14364) (Gerd Hoffmann)
a575af07b8: hw/arm/sbsa-ref: fix typo breaking PCIe IRQs (Graeme Gregory)
5e817ece11: virtio-net: align RSC fields with updated virtio-net header (Yuri 
Benditovich)
b57df52546: nbd: Fix large trim/zero requests (Eric Blake)
921352867b: iotests/028: Add test for cross-base-EOF reads (Max Reitz)
dcf682dd13: block: Fix bdrv_aligned_p*v() for qiov_offset != 0 (Max Reitz)
08550a9940: migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start 
(Vladimir Sementsov-Ogievskiy)
b52a91816e: Update OpenBIOS images to 7f28286f built from submodule. (Mark 
Cave-Ayland)
eca194d43c: libvhost-user: Report descriptor index on panic (Philippe 
Mathieu-Daudé)
d8f0b2bbf4: virtio-pci: Changed vdev to proxy for VirtIO PCI BAR callbacks. 
(Andrew Melnychenko)
58c523563d: intel_iommu: Use correct shift for 256 bits qi descriptor (Liu Yi L)
7c4c30e764: virtio-balloon: Replace free page hinting references to 'report' 
with 'hint' (Alexander Duyck)
595c40575b: linux-headers: update against Linux 5.7-rc3 (Cornelia Huck)
6261aa4ff5: virtio-balloon: always indicate S_DONE when migration fails (David 
Hildenbrand)
7eb63fccf4: virtio-balloon: Add locking to prevent possible race when starting 
hinting (Alexander Duyck)
67808fda37: virtio-balloon: Prevent guest from starting a report when we didn't 
request one (Alexander Duyck)
c16fd8a2bb: qdev: Fix device_add DRIVER,help to print to monitor (Markus 
Armbruster)
d2581f25ec: tests: tpm: Skip over pcrUpdateCounter byte in result comparison 
(Stefan Berger)
42e9a42083: tpm: tpm_spapr: Exit on TPM backend failures (Stefan Berger)
2f783fb459: target/hppa: Free some temps in do_sub (Richard Henderson)
d7fab184e9: hw/sd/sdcard: Do not switch to ReceivingData if address is invalid 
(Philippe Mathieu-Daudé)
c8966bff5f: hw/sd/sdcard: Update coding style to make checkpatch.pl happy 
(Philippe Mathieu-Daudé)
e569ca39fa: hw/sd/sdcard: Do not allow invalid SD card sizes (Philippe 
Mathieu-Daudé)
6a34f7752f: hw/sd/sdcard: Simplify realize() a bit (Philippe Mathieu-Daudé)
557980ba70: hw/sd/sdcard: Restrict Class 6 commands to SCSD cards (Philippe 
Mathieu-Daudé)
b6f4d5bf20: tests/acceptance/boot_linux: Expand SD card image to power of 2 
(Philippe Mathieu-Daudé)
bc67d011c3: tests/acceptance: refactor boot_linux_console test to allow code 
reuse (Pavel Dovgalyuk)
5bdcc0f3ed: tests/acceptance: refactor boot_linux to allow code reuse (Pavel 
Dovgalyuk)
f756254e0a: tests/acceptance: allow console interaction with specific VMs 
(Pavel Dovgalyuk)
7b41093d14: tests/acceptance/boot_linux: Tag tests using a SD card with 
'device:sd' (Philippe Mathieu-Daudé)
acb988e20c: docs/orangepi: Add instructions for resizing SD image to power of 
two (Niek Linnenbank)
3ddb01cd14: qga: Use qemu_get_host_name() instead of g_get_host_name() (Michal 
Privoznik)
ad1169e23c: util: Introduce qemu_get_host_name() (Michal Privoznik)
a799013e71: qga: fix assert regression on guest-shutdown (Marc-André Lureau)
54e74a45a5: chardev/tcp: Fix error message double free error (lichun)
9380cb9de2: nbd: Avoid off-by-one in long export name truncation (Eric Blake)
b3e49baa79: usb/dev-mtp: Fix Error double free after inotify failure (Markus 
Armbruster)
558905635b: error: Use error_reportf_err() where appropriate (Markus Armbruster)
b17b3c209b: net/virtio: Fix failover_replug_primary() return value regression 
(Markus Armbruster)
7a1f17a51e: hw/audio/gus: Fix registers 32-bit access (Allan Peramaki)
0aad2a5255: virtiofsd: Whitelist fchmod (Max Reitz)
0fd3972e1c: hw/net/e1000e: Do not abort() on invalid PSRCTL register value 
(Philippe Mathieu-Daudé)
045849b078: hw/display/artist: Unbreak size mismatch memory accesses (Helge 
Deller)
974b857df6: acpi: accept byte and word access to core ACPI registers (Michael 
Tokarev)
782e94132a: xhci: fix

[PATCH v3 5/5] sev/i386: Enable an SEV-ES guest based on SEV policy

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

Update the sev_es_enabled() function return value to be based on the SEV
policy that has been specified. SEV-ES is enabled if SEV is enabled and
the SEV-ES policy bit is set in the policy object.

Signed-off-by: Tom Lendacky 
---
 target/i386/sev.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/target/i386/sev.c b/target/i386/sev.c
index 6ddefc65fa..bcaadaa2f9 100644
--- a/target/i386/sev.c
+++ b/target/i386/sev.c
@@ -70,6 +70,8 @@ struct SevGuestState {
 #define DEFAULT_GUEST_POLICY0x1 /* disable debug */
 #define DEFAULT_SEV_DEVICE  "/dev/sev"
 
+#define GUEST_POLICY_SEV_ES_BIT (1 << 2)
+
 /* SEV Information Block GUID = 00f771de-1a7e-4fcb-890e-68c77e2fb44e */
 #define SEV_INFO_BLOCK_GUID \
 "\xde\x71\xf7\x00\x7e\x1a\xcb\x4f\x89\x0e\x68\xc7\x7e\x2f\xb4\x4e"
@@ -375,7 +377,7 @@ sev_enabled(void)
 bool
 sev_es_enabled(void)
 {
-return false;
+return sev_enabled() && (sev_guest->policy & GUEST_POLICY_SEV_ES_BIT);
 }
 
 uint64_t
-- 
2.28.0

[PATCH v3 4/5] sev/i386: Don't allow a system reset under an SEV-ES guest

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

An SEV-ES guest does not allow register state to be altered once it has
been measured. When a SEV-ES guest issues a reboot command, Qemu will
reset the vCPU state and resume the guest. This will cause failures under
SEV-ES, so prevent that from occurring.

Signed-off-by: Tom Lendacky 
---
 accel/kvm/kvm-all.c   | 9 +
 include/sysemu/cpus.h | 2 ++
 include/sysemu/hw_accel.h | 5 +
 include/sysemu/kvm.h  | 2 ++
 softmmu/cpus.c| 5 +
 softmmu/vl.c  | 5 -
 6 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index 20725b0368..63153b6e53 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -2388,6 +2388,15 @@ void kvm_flush_coalesced_mmio_buffer(void)
 s->coalesced_flush_in_progress = false;
 }
 
+bool kvm_cpu_check_resettable(void)
+{
+/*
+ * If we have a valid reset vector override, then SEV-ES is active
+ * and the CPU can't be reset.
+ */
+return !kvm_state->reset_valid;
+}
+
 static void do_kvm_cpu_synchronize_state(CPUState *cpu, run_on_cpu_data arg)
 {
 if (!cpu->vcpu_dirty) {
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 3c1da6a018..6d688c757f 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -24,6 +24,8 @@ void dump_drift_info(void);
 void qemu_cpu_kick_self(void);
 void qemu_timer_notify_cb(void *opaque, QEMUClockType type);
 
+bool cpu_is_resettable(void);
+
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
 void cpu_synchronize_all_post_init(void);
diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index e128f8b06b..8b4536e7ae 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -17,6 +17,11 @@
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
 
+static inline bool cpu_check_resettable(void)
+{
+return kvm_enabled() ? kvm_cpu_check_resettable() : true;
+}
+
 static inline void cpu_synchronize_state(CPUState *cpu)
 {
 if (kvm_enabled()) {
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index f74cfa85ab..eb94bbbff9 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -494,6 +494,8 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void 
*ram_addr,
 
 #endif /* NEED_CPU_H */
 
+bool kvm_cpu_check_resettable(void);
+
 void kvm_cpu_synchronize_state(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
diff --git a/softmmu/cpus.c b/softmmu/cpus.c
index a802e899ab..32f286643f 100644
--- a/softmmu/cpus.c
+++ b/softmmu/cpus.c
@@ -927,6 +927,11 @@ void hw_error(const char *fmt, ...)
 abort();
 }
 
+bool cpu_is_resettable(void)
+{
+return cpu_check_resettable();
+}
+
 void cpu_synchronize_all_states(void)
 {
 CPUState *cpu;
diff --git a/softmmu/vl.c b/softmmu/vl.c
index 4eb9d1f7fd..422fbb1650 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -1475,7 +1475,10 @@ void qemu_system_guest_crashloaded(GuestPanicInformation 
*info)
 
 void qemu_system_reset_request(ShutdownCause reason)
 {
-if (no_reboot && reason != SHUTDOWN_CAUSE_SUBSYSTEM_RESET) {
+if (!cpu_is_resettable()) {
+error_report("cpus are not resettable, terminating");
+shutdown_requested = reason;
+} else if (no_reboot && reason != SHUTDOWN_CAUSE_SUBSYSTEM_RESET) {
 shutdown_requested = reason;
 } else {
 reset_requested = reason;
-- 
2.28.0

Re: [PATCH 1/5] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Richard Henderson

On 9/15/20 2:27 PM, Eduardo Habkost wrote:
> On Tue, Sep 15, 2020 at 01:51:48PM -0700, Richard Henderson wrote:
>> On 9/15/20 1:19 PM, Eduardo Habkost wrote:
>>> Once we move to C11, we can just use max_align_t.
>>
>> Yes.
>>
>>> While we don't move to C11, why not just use
>>>   __alignof__(union { long l; void *p; double d; long double ld;})
>>> ?
>>
>> For i386, this is 4.
> 
> Is i386-linux the only case where there are additional alignment
> guarantees not covered by C99?

I think so.

> I would prefer a i386-linux-specific #ifdef for that case instead
> of guessing based on undocumented libc internals.

I was thinking abi, not internals.


r~

[PATCH v3 2/5] sev/i386: Require in-kernel irqchip support for SEV-ES guests

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

In prep for AP booting, require the use of in-kernel irqchip support. This
lessens the Qemu support burden required to boot APs.

Signed-off-by: Tom Lendacky 
---
 target/i386/sev.c | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/target/i386/sev.c b/target/i386/sev.c
index 6c9cd0854b..5055b1fe00 100644
--- a/target/i386/sev.c
+++ b/target/i386/sev.c
@@ -776,6 +776,12 @@ sev_guest_init(const char *id)
 sev->api_minor = status.api_minor;
 
 if (sev_es_enabled()) {
+if (!kvm_kernel_irqchip_allowed()) {
+error_report("%s: SEV-ES guests require in-kernel irqchip support",
+ __func__);
+goto err;
+}
+
 if (!(status.flags & SEV_STATUS_FLAGS_CONFIG_ES)) {
 error_report("%s: guest policy requires SEV-ES, but "
  "host SEV-ES support unavailable",
-- 
2.28.0

[PATCH v3 3/5] sev/i386: Allow AP booting under SEV-ES

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

When SEV-ES is enabled, it is not possible modify the guests register
state after it has been initially created, encrypted and measured.

Normally, an INIT-SIPI-SIPI request is used to boot the AP. However, the
hypervisor cannot emulate this because it cannot update the AP register
state. For the very first boot by an AP, the reset vector CS segment
value and the EIP value must be programmed before the register has been
encrypted and measured.

Signed-off-by: Tom Lendacky 
---
 accel/kvm/kvm-all.c| 64 ++
 accel/stubs/kvm-stub.c |  5 
 hw/i386/pc_sysfw.c | 10 ++-
 include/sysemu/kvm.h   | 16 +++
 include/sysemu/sev.h   |  3 ++
 target/i386/kvm.c  |  2 ++
 target/i386/sev.c  | 51 +
 7 files changed, 150 insertions(+), 1 deletion(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index 63ef6af9a1..20725b0368 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -39,6 +39,7 @@
 #include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/irq.h"
+#include "sysemu/kvm.h"
 #include "sysemu/sev.h"
 #include "qapi/visitor.h"
 #include "qapi/qapi-types-common.h"
@@ -120,6 +121,12 @@ struct KVMState
 /* memory encryption */
 void *memcrypt_handle;
 int (*memcrypt_encrypt_data)(void *handle, uint8_t *ptr, uint64_t len);
+int (*memcrypt_save_reset_vector)(void *handle, void *flash_ptr,
+  uint64_t flash_size, uint32_t *addr);
+
+unsigned int reset_cs;
+unsigned int reset_ip;
+bool reset_valid;
 
 /* For "info mtree -f" to tell if an MR is registered in KVM */
 int nr_as;
@@ -239,6 +246,62 @@ int kvm_memcrypt_encrypt_data(uint8_t *ptr, uint64_t len)
 return 1;
 }
 
+void kvm_memcrypt_set_reset_vector(CPUState *cpu)
+{
+X86CPU *x86;
+CPUX86State *env;
+
+/* Only update if we have valid reset information */
+if (!kvm_state->reset_valid) {
+return;
+}
+
+/* Do not update the BSP reset state */
+if (cpu->cpu_index == 0) {
+return;
+}
+
+x86 = X86_CPU(cpu);
+env = >env;
+
+cpu_x86_load_seg_cache(env, R_CS, 0xf000, kvm_state->reset_cs, 0x,
+   DESC_P_MASK | DESC_S_MASK | DESC_CS_MASK |
+   DESC_R_MASK | DESC_A_MASK);
+
+env->eip = kvm_state->reset_ip;
+}
+
+int kvm_memcrypt_save_reset_vector(void *flash_ptr, uint64_t flash_size)
+{
+CPUState *cpu;
+uint32_t addr;
+int ret;
+
+if (kvm_memcrypt_enabled() &&
+kvm_state->memcrypt_save_reset_vector) {
+
+addr = 0;
+ret = kvm_state->memcrypt_save_reset_vector(kvm_state->memcrypt_handle,
+flash_ptr, flash_size,
+);
+if (ret) {
+return ret;
+}
+
+if (addr) {
+kvm_state->reset_cs = addr & 0x;
+kvm_state->reset_ip = addr & 0x;
+kvm_state->reset_valid = true;
+
+CPU_FOREACH(cpu) {
+kvm_memcrypt_set_reset_vector(cpu);
+}
+}
+}
+
+return 0;
+}
+
 /* Called with KVMMemoryListener.slots_lock held */
 static KVMSlot *kvm_get_free_slot(KVMMemoryListener *kml)
 {
@@ -2193,6 +2256,7 @@ static int kvm_init(MachineState *ms)
 }
 
 kvm_state->memcrypt_encrypt_data = sev_encrypt_data;
+kvm_state->memcrypt_save_reset_vector = sev_es_save_reset_vector;
 }
 
 ret = kvm_arch_init(ms, s);
diff --git a/accel/stubs/kvm-stub.c b/accel/stubs/kvm-stub.c
index 82f118d2df..3aece9b513 100644
--- a/accel/stubs/kvm-stub.c
+++ b/accel/stubs/kvm-stub.c
@@ -114,6 +114,11 @@ int kvm_memcrypt_encrypt_data(uint8_t *ptr, uint64_t len)
   return 1;
 }
 
+int kvm_memcrypt_save_reset_vector(void *flash_ptr, uint64_t flash_size)
+{
+return -ENOSYS;
+}
+
 #ifndef CONFIG_USER_ONLY
 int kvm_irqchip_add_msi_route(KVMState *s, int vector, PCIDevice *dev)
 {
diff --git a/hw/i386/pc_sysfw.c b/hw/i386/pc_sysfw.c
index b6c0822fe3..321ff94261 100644
--- a/hw/i386/pc_sysfw.c
+++ b/hw/i386/pc_sysfw.c
@@ -156,7 +156,8 @@ static void pc_system_flash_map(PCMachineState *pcms,
 PFlashCFI01 *system_flash;
 MemoryRegion *flash_mem;
 void *flash_ptr;
-int ret, flash_size;
+uint64_t flash_size;
+int ret;
 
 assert(PC_MACHINE_GET_CLASS(pcms)->pci_enabled);
 
@@ -204,6 +205,13 @@ static void pc_system_flash_map(PCMachineState *pcms,
 if (kvm_memcrypt_enabled()) {
 flash_ptr = memory_region_get_ram_ptr(flash_mem);
 flash_size = memory_region_size(flash_mem);
+
+ret = kvm_memcrypt_save_reset_vector(flash_ptr, flash_size);
+if (ret) {
+error_report("failed to locate and/or save reset vector");
+exit(1);
+}
+
 ret =

[PATCH v3 1/5] sev/i386: Add initial support for SEV-ES

2020-09-15 Thread Tom Lendacky

From: Tom Lendacky 

Provide initial support for SEV-ES. This includes creating a function to
indicate the guest is an SEV-ES guest (which will return false until all
support is in place), performing the proper SEV initialization and
ensuring that the guest CPU state is measured as part of the launch.

Co-developed-by: Jiri Slaby 
Signed-off-by: Jiri Slaby 
Signed-off-by: Tom Lendacky 
---
 target/i386/cpu.c  |  1 +
 target/i386/sev-stub.c |  5 +
 target/i386/sev.c  | 46 --
 target/i386/sev_i386.h |  1 +
 4 files changed, 51 insertions(+), 2 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 588f32e136..bbbe581d35 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5969,6 +5969,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, 
uint32_t count,
 break;
 case 0x801F:
 *eax = sev_enabled() ? 0x2 : 0;
+*eax |= sev_es_enabled() ? 0x8 : 0;
 *ebx = sev_get_cbit_position();
 *ebx |= sev_get_reduced_phys_bits() << 6;
 *ecx = 0;
diff --git a/target/i386/sev-stub.c b/target/i386/sev-stub.c
index 88e3f39a1e..040ac90563 100644
--- a/target/i386/sev-stub.c
+++ b/target/i386/sev-stub.c
@@ -49,3 +49,8 @@ SevCapability *sev_get_capabilities(Error **errp)
 error_setg(errp, "SEV is not available in this QEMU");
 return NULL;
 }
+
+bool sev_es_enabled(void)
+{
+return false;
+}
diff --git a/target/i386/sev.c b/target/i386/sev.c
index c3ecf86704..6c9cd0854b 100644
--- a/target/i386/sev.c
+++ b/target/i386/sev.c
@@ -359,6 +359,12 @@ sev_enabled(void)
 return !!sev_guest;
 }
 
+bool
+sev_es_enabled(void)
+{
+return false;
+}
+
 uint64_t
 sev_get_me_mask(void)
 {
@@ -578,6 +584,22 @@ sev_launch_update_data(SevGuestState *sev, uint8_t *addr, 
uint64_t len)
 return ret;
 }
 
+static int
+sev_launch_update_vmsa(SevGuestState *sev)
+{
+int ret, fw_error;
+
+ret = sev_ioctl(sev->sev_fd, KVM_SEV_LAUNCH_UPDATE_VMSA, NULL, _error);
+if (ret) {
+error_report("%s: LAUNCH_UPDATE_VMSA ret=%d fw_error=%d '%s'",
+__func__, ret, fw_error, fw_error_to_str(fw_error));
+goto err;
+}
+
+err:
+return ret;
+}
+
 static void
 sev_launch_get_measure(Notifier *notifier, void *unused)
 {
@@ -590,6 +612,14 @@ sev_launch_get_measure(Notifier *notifier, void *unused)
 return;
 }
 
+if (sev_es_enabled()) {
+/* measure all the VM save areas before getting launch_measure */
+ret = sev_launch_update_vmsa(sev);
+if (ret) {
+exit(1);
+}
+}
+
 measurement = g_new0(struct kvm_sev_launch_measure, 1);
 
 /* query the measurement blob length */
@@ -684,7 +714,7 @@ sev_guest_init(const char *id)
 {
 SevGuestState *sev;
 char *devname;
-int ret, fw_error;
+int ret, fw_error, cmd;
 uint32_t ebx;
 uint32_t host_cbitpos;
 struct sev_user_data_status status = {};
@@ -745,8 +775,20 @@ sev_guest_init(const char *id)
 sev->api_major = status.api_major;
 sev->api_minor = status.api_minor;
 
+if (sev_es_enabled()) {
+if (!(status.flags & SEV_STATUS_FLAGS_CONFIG_ES)) {
+error_report("%s: guest policy requires SEV-ES, but "
+ "host SEV-ES support unavailable",
+ __func__);
+goto err;
+}
+cmd = KVM_SEV_ES_INIT;
+} else {
+cmd = KVM_SEV_INIT;
+}
+
 trace_kvm_sev_init();
-ret = sev_ioctl(sev->sev_fd, KVM_SEV_INIT, NULL, _error);
+ret = sev_ioctl(sev->sev_fd, cmd, NULL, _error);
 if (ret) {
 error_report("%s: failed to initialize ret=%d fw_error=%d '%s'",
  __func__, ret, fw_error, fw_error_to_str(fw_error));
diff --git a/target/i386/sev_i386.h b/target/i386/sev_i386.h
index 4db6960f60..4f9a5e9b21 100644
--- a/target/i386/sev_i386.h
+++ b/target/i386/sev_i386.h
@@ -29,6 +29,7 @@
 #define SEV_POLICY_SEV  0x20
 
 extern bool sev_enabled(void);
+extern bool sev_es_enabled(void);
 extern uint64_t sev_get_me_mask(void);
 extern SevInfo *sev_get_info(void);
 extern uint32_t sev_get_cbit_position(void);
-- 
2.28.0

Re: [PATCH 1/5] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Eduardo Habkost

On Tue, Sep 15, 2020 at 01:51:48PM -0700, Richard Henderson wrote:
> On 9/15/20 1:19 PM, Eduardo Habkost wrote:
> > Once we move to C11, we can just use max_align_t.
> 
> Yes.
> 
> > While we don't move to C11, why not just use
> >   __alignof__(union { long l; void *p; double d; long double ld;})
> > ?
> 
> For i386, this is 4.

Is i386-linux the only case where there are additional alignment
guarantees not covered by C99?

I would prefer a i386-linux-specific #ifdef for that case instead
of guessing based on undocumented libc internals.

-- 
Eduardo

[REPORT] Nightly Performance Tests - Tuesday, September 15, 2020

2020-09-15 Thread Ahmed Karaman


Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Host Memory  : 15.49 GB

Start Time (UTC) : 2020-09-15 20:40:01
End Time (UTC)   : 2020-09-15 21:12:43
Execution Time   : 0:32:41.290408

Status   : SUCCESS

Note:
Changes denoted by '-' are less than 0.01%.


SUMMARY REPORT - COMMIT de39a045

AVERAGE RESULTS

Target  Instructions  Latest  v5.1.0
--    --  --
aarch642 158 524 492   - +1.704%
alpha  1 914 934 541   - +3.521%
arm8 076 523 798   - +2.308%
hppa   4 261 683 298   - +3.163%
m68k   2 690 291 552   - +7.134%
mips   1 861 895 402   - +2.484%
mipsel 2 008 239 436   - +2.676%
mips64 1 918 601 151   - +2.816%
mips64el   2 051 554 246   - +3.025%
ppc2 480 178 421   -  +3.11%
ppc64  2 576 714 241   - +3.143%
ppc64le2 558 832 167   - +3.171%
riscv641 406 692 488   - +2.649%
s390x  3 158 132 301   - +3.118%
sh42 364 605 623   - +3.341%
sparc643 318 701 656   - +3.855%
x86_64 1 775 944 131   - +2.167%


   DETAILED RESULTS

Test Program: dijkstra_double

Target  Instructions  Latest  v5.1.0
--    --  --
aarch643 062 758 181   -  +1.43%
alpha  3 191 832 563   - +3.695%
arm   16 357 287 709   - +2.348%
hppa   7 228 398 607   - +3.086%
m68k   4 294 056 637   - +9.693%
mips   3 051 310 255   - +2.423%
mipsel 3 231 548 335   -  +2.87%
mips64 3 245 791 692   - +2.595%
mips64el   3 414 214 247   - +3.021%
ppc4 914 561 218   -  +4.74%
ppc64  5 098 152 776   - +4.565%
ppc64le5 082 399 487   -  +4.58%
riscv642 192 276 027   - +1.955%
s390x  4 584 581 806   - +2.897%
sh43 949 197 215   - +3.468%
sparc644 586 107 304   - +4.235%
x86_64 2 484 250 887   - +1.757%


Test Program: dijkstra_int32

Target  Instructions  Latest  v5.1.0
--    --  --
aarch642 210 370 607   - +1.502%
alpha  1 494 098 562   - +2.148%
arm8 263 040 235   - +2.667%
hppa   5 207 316 726   - +3.047%
m68k   1 725 877 759   - +2.528%
mips   1 495 100 498   - +1.483%
mipsel 1 497 165 574   -  +1.48%
mips64 1 715 397 024   - +1.892%
mips64el   1 695 210 741   - +1.909%
ppc2 014 604 657   - +1.822%
ppc64  2 206 271 827   - +2.139%
ppc64le2 197 979 437   - +2.145%
riscv641 354 885 081   - +2.394%
s390x  2 916 086 983   - +1.236%
sh41 990 693 213   - +2.678%
sparc642 874 141 483   - +3.827%
x86_64 1 554 140 016   -  +2.13%


Test Program: matmult_double

Target  Instructions  Latest  v5.1.0
--    --  --
aarch641 412 427 654   - +0.313%
alpha  3 233 959 954   - +7.472%
arm8 545 300 488   -  +1.09%
hppa   3 483 527 384   - +4.466%
m68k   3 919 108 398   -+18.433%
mips   2 344 640 021   - +4.085%
mipsel 3 329 921 331   - +5.178%
mips64 2 359 007 385   - +4.074%

[Bug 1892604] Re: qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion `addr <= GINTSTS2' failed.

2020-09-15 Thread Petunia

Nope:

qemu-system-arm -M raspi2 -m 1G -dtb bcm2709-rpi-2-b.dtb -kernel
kernel7.img -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200
dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 dwc_otg.fiq_fsm_enable=0" -sd
2016-05-27-raspbian-jessie.img -smp 4 -serial stdio -display none

raspberrypi login: qemu-system-arm: 
/build/qemu/src/qemu-5.1.0/hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion 
`addr <= GINTSTS2' failed.
[1]65608 abort (core dumped)  qemu-system-arm -M raspi2 -m 1G -dtb 
bcm2709-rpi-2-b.dtb -kernel kernel7.img

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1892604

Title:
  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

Status in QEMU:
  New

Bug description:
  When trying to run the 2016-05-27 Raspbian image on the emulated
  raspi2 platform, the system boots but shortly after the login prompt
  QEMU (master; commit ID ca489cd037e4d50dc6c40570a167504ad7e5a521) dies
  with:

  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

  Steps to reproduce:

  1. Get the image: wget
  
http://downloads.raspberrypi.org/raspbian/images/raspbian-2016-05-31/2016-05-27
  -raspbian-jessie.zip

  2. Extract the kernel image and DTB:

  sudo losetup -f --show -P 2016-05-27-raspbian-jessie.img
  sudo mkdir /mnt/rpi
  sudo mount /dev/loop11p1 /mnt/rpi/
  cp /mnt/rpi/kernel7.img . 



  cp /mnt/rpi/bcm2709-rpi-2-b.dtb . 



  sudo umount /mnt/rpi 
  sudo losetup -d /dev/loop11 

  3. Run QEMU:
  qemu-system-arm -M raspi2 -m 1G -dtb bcm2709-rpi-2-b.dtb -kernel kernel7.img 
-append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 
root=/dev/mmcblk0p2" -sd 2016-05-27-raspbian-jessie.img -smp 4 -serial stdio 
-display none

  A few seconds after the login prompt is displayed, QEMU will exit with
  the assertion failure.

  I also tried changing all of the asserts to if statements that (for
  MMIO reads) returned 0 and (for writes) just returned, but this
  resulted in a non-responsive system.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1892604/+subscriptions

Re: [PATCH 1/5] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Richard Henderson

On 9/15/20 1:19 PM, Eduardo Habkost wrote:
> Once we move to C11, we can just use max_align_t.

Yes.

> While we don't move to C11, why not just use
>   __alignof__(union { long l; void *p; double d; long double ld;})
> ?

For i386, this is 4.


r~

Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result

2020-09-15 Thread Klaus Jensen

On Sep 15 20:44, Dmitry Fomichev wrote:
> > -Original Message-
> > From: Klaus Jensen 
> > Sent: Tuesday, September 15, 2020 3:56 PM
> > To: Dmitry Fomichev 
> > Cc: Fam Zheng ; Kevin Wolf ;
> > Damien Le Moal ; qemu-bl...@nongnu.org;
> > Niklas Cassel ; Klaus Jensen
> > ; qemu-devel@nongnu.org; Alistair Francis
> > ; Keith Busch ; Philippe
> > Mathieu-Daudé ; Matias Bjorling
> > 
> > Subject: Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result
> > 
> > On Sep 15 18:56, Dmitry Fomichev wrote:
> > > > -Original Message-
> > > > From: Klaus Jensen 
> > > > Sent: Tuesday, September 15, 2020 3:37 AM
> > > > To: Dmitry Fomichev 
> > > > Cc: Keith Busch ; Klaus Jensen
> > > > ; Kevin Wolf ; Philippe
> > > > Mathieu-Daudé ; Maxim Levitsky
> > > > ; Fam Zheng ; Niklas Cassel
> > > > ; Damien Le Moal
> > ;
> > > > qemu-bl...@nongnu.org; qemu-devel@nongnu.org; Alistair Francis
> > > > ; Matias Bjorling 
> > > > Subject: Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result
> > > >
> > > > On Sep 14 07:14, Dmitry Fomichev wrote:
> > > > > From: Ajay Joshi 
> > > > >
> > > > > A new write command, Zone Append, is added as a part of Zoned
> > > > > Namespace Command Set. Upon successful completion of this
> > command,
> > > > > the controller returns the start LBA of the performed write operation
> > > > > in cqe.result field. Therefore, the maximum size of this variable
> > > > > needs to be changed from 32 to 64 bit, consuming the reserved 32 bit
> > > > > field that follows the result in CQE struct. Since the existing
> > > > > commands are expected to return a 32 bit LE value, two separate
> > > > > variables, result32 and result64, are now kept in a union.
> > > > >
> > > > > Signed-off-by: Ajay Joshi 
> > > > > Signed-off-by: Dmitry Fomichev 
> > > > > Reviewed-by: Klaus Jensen 
> > > >
> > > > I know that I R-b'ed this, but can this be moved to the namespace types
> > > > patch, since that is the TP that changes this.
> > >
> > > You probably meant the ZNS patch since result64 is first used there to
> > return
> > > ZA starting data LBA. Sure, I can move this stuff to that patch.
> > >
> > 
> > No, I actually did mean the NST patch since TP 4056 is the TP that
> > "unreserves" dw1 in the CQE.
> 
> It is not necessary to change it in NST patch since result64 field is not used
> in that patch. But if you insist, I can move it to NST patch :)

You are right of course, but since it is a change to the "spec" related
data structures that go into include/block/nvme.h, I think it belongs in
"hw/block/nvme: Introduce the Namespace Types definitions".


signature.asc
Description: PGP signature

Re: [PATCH v3 14/15] hw/block/nvme: Use zone metadata file for persistence

2020-09-15 Thread Klaus Jensen

On Sep 15 20:44, Dmitry Fomichev wrote:
> > -Original Message-
> > From: Klaus Jensen 
> > Sent: Tuesday, September 15, 2020 3:10 PM
> > To: Dmitry Fomichev 
> > Cc: Keith Busch ; Klaus Jensen
> > ; Kevin Wolf ; Philippe
> > Mathieu-Daudé ; Maxim Levitsky
> > ; Fam Zheng ; Niklas Cassel
> > ; Damien Le Moal ;
> > qemu-bl...@nongnu.org; qemu-devel@nongnu.org; Alistair Francis
> > ; Matias Bjorling 
> > Subject: Re: [PATCH v3 14/15] hw/block/nvme: Use zone metadata file for
> > persistence
> > 
> > On Sep 14 07:14, Dmitry Fomichev wrote:
> > > A ZNS drive that is emulated by this module is currently initialized
> > > with all zones Empty upon startup. However, actual ZNS SSDs save the
> > > state and condition of all zones in their internal NVRAM in the event
> > > of power loss. When such a drive is powered up again, it closes or
> > > finishes all zones that were open at the moment of shutdown. Besides
> > > that, the write pointer position as well as the state and condition
> > > of all zones is preserved across power-downs.
> > >
> > > This commit adds the capability to have a persistent zone metadata
> > > to the device. The new optional module property, "zone_file",
> > > is introduced. If added to the command line, this property specifies
> > > the name of the file that stores the zone metadata. If "zone_file" is
> > > omitted, the device will be initialized with all zones empty, the same
> > > as before.
> > >
> > > If zone metadata is configured to be persistent, then zone descriptor
> > > extensions also persist across controller shutdowns.
> > >
> > > Signed-off-by: Dmitry Fomichev 
> > 
> > This doesn't build on mingw.
> 
> Thanks for letting me know. I'll try to look into this. Do you cross-compile
> with mingw64?

Yup,

make docker-test-mingw@fedora



signature.asc
Description: PGP signature

RE: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result

2020-09-15 Thread Dmitry Fomichev

> -Original Message-
> From: Klaus Jensen 
> Sent: Tuesday, September 15, 2020 3:56 PM
> To: Dmitry Fomichev 
> Cc: Fam Zheng ; Kevin Wolf ;
> Damien Le Moal ; qemu-bl...@nongnu.org;
> Niklas Cassel ; Klaus Jensen
> ; qemu-devel@nongnu.org; Alistair Francis
> ; Keith Busch ; Philippe
> Mathieu-Daudé ; Matias Bjorling
> 
> Subject: Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result
> 
> On Sep 15 18:56, Dmitry Fomichev wrote:
> > > -Original Message-
> > > From: Klaus Jensen 
> > > Sent: Tuesday, September 15, 2020 3:37 AM
> > > To: Dmitry Fomichev 
> > > Cc: Keith Busch ; Klaus Jensen
> > > ; Kevin Wolf ; Philippe
> > > Mathieu-Daudé ; Maxim Levitsky
> > > ; Fam Zheng ; Niklas Cassel
> > > ; Damien Le Moal
> ;
> > > qemu-bl...@nongnu.org; qemu-devel@nongnu.org; Alistair Francis
> > > ; Matias Bjorling 
> > > Subject: Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result
> > >
> > > On Sep 14 07:14, Dmitry Fomichev wrote:
> > > > From: Ajay Joshi 
> > > >
> > > > A new write command, Zone Append, is added as a part of Zoned
> > > > Namespace Command Set. Upon successful completion of this
> command,
> > > > the controller returns the start LBA of the performed write operation
> > > > in cqe.result field. Therefore, the maximum size of this variable
> > > > needs to be changed from 32 to 64 bit, consuming the reserved 32 bit
> > > > field that follows the result in CQE struct. Since the existing
> > > > commands are expected to return a 32 bit LE value, two separate
> > > > variables, result32 and result64, are now kept in a union.
> > > >
> > > > Signed-off-by: Ajay Joshi 
> > > > Signed-off-by: Dmitry Fomichev 
> > > > Reviewed-by: Klaus Jensen 
> > >
> > > I know that I R-b'ed this, but can this be moved to the namespace types
> > > patch, since that is the TP that changes this.
> >
> > You probably meant the ZNS patch since result64 is first used there to
> return
> > ZA starting data LBA. Sure, I can move this stuff to that patch.
> >
> 
> No, I actually did mean the NST patch since TP 4056 is the TP that
> "unreserves" dw1 in the CQE.

It is not necessary to change it in NST patch since result64 field is not used
in that patch. But if you insist, I can move it to NST patch :)

RE: [PATCH v3 14/15] hw/block/nvme: Use zone metadata file for persistence

2020-09-15 Thread Dmitry Fomichev

> -Original Message-
> From: Klaus Jensen 
> Sent: Tuesday, September 15, 2020 3:10 PM
> To: Dmitry Fomichev 
> Cc: Keith Busch ; Klaus Jensen
> ; Kevin Wolf ; Philippe
> Mathieu-Daudé ; Maxim Levitsky
> ; Fam Zheng ; Niklas Cassel
> ; Damien Le Moal ;
> qemu-bl...@nongnu.org; qemu-devel@nongnu.org; Alistair Francis
> ; Matias Bjorling 
> Subject: Re: [PATCH v3 14/15] hw/block/nvme: Use zone metadata file for
> persistence
> 
> On Sep 14 07:14, Dmitry Fomichev wrote:
> > A ZNS drive that is emulated by this module is currently initialized
> > with all zones Empty upon startup. However, actual ZNS SSDs save the
> > state and condition of all zones in their internal NVRAM in the event
> > of power loss. When such a drive is powered up again, it closes or
> > finishes all zones that were open at the moment of shutdown. Besides
> > that, the write pointer position as well as the state and condition
> > of all zones is preserved across power-downs.
> >
> > This commit adds the capability to have a persistent zone metadata
> > to the device. The new optional module property, "zone_file",
> > is introduced. If added to the command line, this property specifies
> > the name of the file that stores the zone metadata. If "zone_file" is
> > omitted, the device will be initialized with all zones empty, the same
> > as before.
> >
> > If zone metadata is configured to be persistent, then zone descriptor
> > extensions also persist across controller shutdowns.
> >
> > Signed-off-by: Dmitry Fomichev 
> 
> This doesn't build on mingw.

Thanks for letting me know. I'll try to look into this. Do you cross-compile
with mingw64?

> 
> > ---
> >  hw/block/nvme.c | 370
> +---
> >  hw/block/nvme.h |  37 +
> >  2 files changed, 386 insertions(+), 21 deletions(-)
> >
> > diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> > index b49ae83dd5..41f4c0dacd 100644
> > --- a/hw/block/nvme.c
> > +++ b/hw/block/nvme.c
> > @@ -3429,7 +3557,188 @@ static int nvme_init_zone_meta(NvmeCtrl *n,
> NvmeNamespace *ns,
> >  return 0;
> >  }
> >
> > -static void nvme_zoned_init_ctrl(NvmeCtrl *n, Error **errp)
> > +static int nvme_open_zone_file(NvmeCtrl *n, bool *init_meta)
> > +{
> > +struct stat statbuf;
> > +size_t fsize;
> > +int ret;
> > +
> > +ret = stat(n->params.zone_file, );
> > +if (ret && errno == ENOENT) {
> > +*init_meta = true;
> > +} else if (!S_ISREG(statbuf.st_mode)) {
> > +fprintf(stderr, "%s is not a regular file\n", strerror(errno));
> > +return -1;
> > +}
> > +
> > +n->zone_file_fd = open(n->params.zone_file,
> > +   O_RDWR | O_LARGEFILE | O_BINARY | O_CREAT, 644);
> 
> mode is wrong - I think you meant for it to be octal.

Nice catch, needs to be 0644...

Re: [PATCH 1/5] qom: Allow objects to be allocated with increased alignment

2020-09-15 Thread Eduardo Habkost

On Tue, Sep 15, 2020 at 12:09:59PM -0700, Richard Henderson wrote:
> On 9/15/20 11:07 AM, Eduardo Habkost wrote:
> > On Tue, Sep 15, 2020 at 10:46:31AM -0700, Richard Henderson wrote:
> >> It turns out that some hosts have a default malloc alignment less
> >> than that required for vectors.
> >>
> >> We assume that, with compiler annotation on CPUArchState, that we
> >> can properly align the vector portion of the guest state.  Fix the
> >> alignment of the allocation by using qemu_memalloc when required.
> >>
> >> Signed-off-by: Richard Henderson 
> >> ---
> >> Cc: Paolo Bonzini 
> >> Cc: "Daniel P. Berrangé" 
> >> Cc: Eduardo Habkost 
> >> ---
> >>  include/qom/object.h |  4 
> >>  qom/object.c | 16 +---
> >>  2 files changed, 17 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/include/qom/object.h b/include/qom/object.h
> >> index 056f67ab3b..d52d0781a3 100644
> >> --- a/include/qom/object.h
> >> +++ b/include/qom/object.h
> >> @@ -770,6 +770,9 @@ struct Object
> >>   * @instance_size: The size of the object (derivative of #Object).  If
> >>   *   @instance_size is 0, then the size of the object will be the size of 
> >> the
> >>   *   parent object.
> >> + * @instance_align: The required alignment of the object.  If 
> >> @instance_align
> >> + *   is 0, then normal malloc alignment is sufficient; if non-zero, then 
> >> we
> >> + *   must use qemu_memalign for allocation.
> >>   * @instance_init: This function is called to initialize an object.  The 
> >> parent
> >>   *   class will have already been initialized so the type is only 
> >> responsible
> >>   *   for initializing its own members.
> >> @@ -807,6 +810,7 @@ struct TypeInfo
> >>  const char *parent;
> >>  
> >>  size_t instance_size;
> >> +size_t instance_align;
> >>  void (*instance_init)(Object *obj);
> >>  void (*instance_post_init)(Object *obj);
> >>  void (*instance_finalize)(Object *obj);
> >> diff --git a/qom/object.c b/qom/object.c
> >> index 387efb25eb..2e53cb44a6 100644
> >> --- a/qom/object.c
> >> +++ b/qom/object.c
> >> @@ -50,6 +50,7 @@ struct TypeImpl
> >>  size_t class_size;
> >>  
> >>  size_t instance_size;
> >> +size_t instance_align;
> >>  
> >>  void (*class_init)(ObjectClass *klass, void *data);
> >>  void (*class_base_init)(ObjectClass *klass, void *data);
> >> @@ -114,6 +115,7 @@ static TypeImpl *type_new(const TypeInfo *info)
> >>  
> >>  ti->class_size = info->class_size;
> >>  ti->instance_size = info->instance_size;
> >> +ti->instance_align = info->instance_align;
> >>  
> >>  ti->class_init = info->class_init;
> >>  ti->class_base_init = info->class_base_init;
> >> @@ -691,13 +693,21 @@ static void object_finalize(void *data)
> >>  static Object *object_new_with_type(Type type)
> >>  {
> >>  Object *obj;
> >> +size_t size, align;
> >>  
> >>  g_assert(type != NULL);
> >>  type_initialize(type);
> >>  
> >> -obj = g_malloc(type->instance_size);
> >> -object_initialize_with_type(obj, type->instance_size, type);
> >> -obj->free = g_free;
> >> +size = type->instance_size;
> >> +align = type->instance_align;
> >> +if (align) {
> > 
> > If we check for (align > G_MEM_ALIGN) instead, we will be able to
> > set instance_align automatically at OBJECT_DEFINE_TYPE*.
> 
> I agree a value check would be good here, as well as setting this by default.
> 
> As for the value check itself...
> 
> I see that G_MEM_ALIGN isn't actually defined in an interesting or even 
> correct
> way.  E.g. it doesn't take the long double type into account.
> 
> The usual mechanism is
> 
> struct s {
>   char pad;
>   union {
> long l;
> void *p;
> double d;
> long double ld;
>   } u;
> };
> 
> offsetof(s, u)
> 
> since all of these types are required to be taken into account by the system
> malloc.

Due to the existence of G_MEM_ALIGN, I thought GLib alignment
guarantees could be weaker than malloc().  I've just learned that
GLib uses system malloc() since 2.46.

> 
> E.g it doesn't take other host guarantees into account, e.g. i386-linux
> guarantees 16-byte alignment.  This possibly dubious ABI change was made 20+
> years ago with the introduction of SSE and is now set in stone.
> 
> Glibc has a "malloc-alignment.h" internal header that defaults to
> 
>   MIN(2 * sizeof(size_t), __alignof__(long double))
> 
> and is overridden for i386.  Sadly, it doesn't export MALLOC_ALIGNMENT.
> 
> Musl has two different malloc implementations.  One has UNIT = 16; the other
> has SIZE_ALIGN = 4*sizeof(size_t).  Both have a minimum value of 16, and this
> is not target-specific.
> 
> Any further comments on the subject, or should I put together something that
> computes the MAX of the above?

Once we move to C11, we can just use max_align_t.

While we don't move to C11, why not just use
  __alignof__(union { long l; void *p; double d; long double ld;})
?

-- 
Eduardo

[Bug 1892604] Re: qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion `addr <= GINTSTS2' failed.

2020-09-15 Thread Paul Zimmerman

Are you able to modify the -append string to add
"dwc_otg.fiq_fsm_enable=0", and if so does it fix the problem?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1892604

Title:
  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

Status in QEMU:
  New

Bug description:
  When trying to run the 2016-05-27 Raspbian image on the emulated
  raspi2 platform, the system boots but shortly after the login prompt
  QEMU (master; commit ID ca489cd037e4d50dc6c40570a167504ad7e5a521) dies
  with:

  qemu-system-arm: ../hw/usb/hcd-dwc2.c:666: dwc2_glbreg_read: Assertion
  `addr <= GINTSTS2' failed.

  Steps to reproduce:

  1. Get the image: wget
  
http://downloads.raspberrypi.org/raspbian/images/raspbian-2016-05-31/2016-05-27
  -raspbian-jessie.zip

  2. Extract the kernel image and DTB:

  sudo losetup -f --show -P 2016-05-27-raspbian-jessie.img
  sudo mkdir /mnt/rpi
  sudo mount /dev/loop11p1 /mnt/rpi/
  cp /mnt/rpi/kernel7.img . 



  cp /mnt/rpi/bcm2709-rpi-2-b.dtb . 



  sudo umount /mnt/rpi 
  sudo losetup -d /dev/loop11 

  3. Run QEMU:
  qemu-system-arm -M raspi2 -m 1G -dtb bcm2709-rpi-2-b.dtb -kernel kernel7.img 
-append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 
root=/dev/mmcblk0p2" -sd 2016-05-27-raspbian-jessie.img -smp 4 -serial stdio 
-display none

  A few seconds after the login prompt is displayed, QEMU will exit with
  the assertion failure.

  I also tried changing all of the asserts to if statements that (for
  MMIO reads) returned 0 and (for writes) just returned, but this
  resulted in a non-responsive system.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1892604/+subscriptions

Re: [PATCH v8 4/7] scripts: add block-coroutine-wrapper.py

2020-09-15 Thread Vladimir Sementsov-Ogievskiy


15.09.2020 19:44, Vladimir Sementsov-Ogievskiy wrote:

We have a very frequent pattern of creating coroutine from function
with several arguments:

   - create structure to pack parameters
   - create _entry function to call original function taking parameters
 from struct
   - do different magic to handle completion: set ret to NOT_DONE or
 EINPROGRESS or use separate bool field
   - fill the struct and create coroutine from _entry function and this
 struct as a parameter
   - do coroutine enter and BDRV_POLL_WHILE loop

Let's reduce code duplication by generating coroutine wrappers.

This patch adds scripts/block-coroutine-wrapper.py together with some
friends, which will generate functions with declared prototypes marked
by 'generated_co_wrapper' specifier.

The usage of new code generation is as follows:

 1. define somewhere

 int coroutine_fn bdrv_co_NAME(...) {...}

function

 2. declare in some header file

 int generated_co_wrapper bdrv_NAME(...);

function with same list of parameters. (you'll need to include
"block/generated-co-wrapper.h" to get the specifier)

 3. both declarations should be available through block/coroutines.h
header.

 4. add header with generated_co_wrapper declaration into
COROUTINE_HEADERS list in Makefile

Still, no function is now marked, this work is for the following
commit.

Signed-off-by: Vladimir Sementsov-Ogievskiy
---
  docs/devel/block-coroutine-wrapper.rst |  54 +++
  block/block-gen.h  |  49 +++
  include/block/block.h  |  10 ++
  block/meson.build  |   8 ++
  scripts/block-coroutine-wrapper.py | 187 +
  5 files changed, 308 insertions(+)
  create mode 100644 docs/devel/block-coroutine-wrapper.rst
  create mode 100644 block/block-gen.h
  create mode 100755 scripts/block-coroutine-wrapper.py



Also needed:

diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 04773ce076..cb0abe1e69 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -31,3 +31,4 @@ Contents:
reset
s390-dasd-ipl
clocks
+   block-coroutine-wrapper

--
Best regards,
Vladimir

Re: [PATCH v6 0/8] s390: Extended-Length SCCB & DIAGNOSE 0x318

2020-09-15 Thread no-reply

Patchew URL: 
https://patchew.org/QEMU/20200915194416.107460-1-wall...@linux.ibm.com/



Hi,

This series failed build test on FreeBSD host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
if qemu-system-x86_64 --help >/dev/null 2>&1; then
  QEMU=qemu-system-x86_64
elif /usr/libexec/qemu-kvm --help >/dev/null 2>&1; then
  QEMU=/usr/libexec/qemu-kvm
else
  exit 1
fi
make vm-build-freebsd J=21 QEMU=$QEMU
exit 0
=== TEST SCRIPT END ===




The full log is available at
http://patchew.org/logs/20200915194416.107460-1-wall...@linux.ibm.com/testing.FreeBSD/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result

2020-09-15 Thread Klaus Jensen

On Sep 15 18:56, Dmitry Fomichev wrote:
> > -Original Message-
> > From: Klaus Jensen 
> > Sent: Tuesday, September 15, 2020 3:37 AM
> > To: Dmitry Fomichev 
> > Cc: Keith Busch ; Klaus Jensen
> > ; Kevin Wolf ; Philippe
> > Mathieu-Daudé ; Maxim Levitsky
> > ; Fam Zheng ; Niklas Cassel
> > ; Damien Le Moal ;
> > qemu-bl...@nongnu.org; qemu-devel@nongnu.org; Alistair Francis
> > ; Matias Bjorling 
> > Subject: Re: [PATCH v3 01/15] hw/block/nvme: Define 64 bit cqe.result
> > 
> > On Sep 14 07:14, Dmitry Fomichev wrote:
> > > From: Ajay Joshi 
> > >
> > > A new write command, Zone Append, is added as a part of Zoned
> > > Namespace Command Set. Upon successful completion of this command,
> > > the controller returns the start LBA of the performed write operation
> > > in cqe.result field. Therefore, the maximum size of this variable
> > > needs to be changed from 32 to 64 bit, consuming the reserved 32 bit
> > > field that follows the result in CQE struct. Since the existing
> > > commands are expected to return a 32 bit LE value, two separate
> > > variables, result32 and result64, are now kept in a union.
> > >
> > > Signed-off-by: Ajay Joshi 
> > > Signed-off-by: Dmitry Fomichev 
> > > Reviewed-by: Klaus Jensen 
> > 
> > I know that I R-b'ed this, but can this be moved to the namespace types
> > patch, since that is the TP that changes this.
> 
> You probably meant the ZNS patch since result64 is first used there to return
> ZA starting data LBA. Sure, I can move this stuff to that patch.
> 

No, I actually did mean the NST patch since TP 4056 is the TP that
"unreserves" dw1 in the CQE.


signature.asc
Description: PGP signature

[PATCH v6 6/8] s390/sclp: add extended-length sccb support for kvm guest

2020-09-15 Thread Collin Walling

As more features and facilities are added to the Read SCP Info (RSCPI)
response, more space is required to store them. The space used to store
these new features intrudes on the space originally used to store CPU
entries. This means as more features and facilities are added to the
RSCPI response, less space can be used to store CPU entries.

With the Extended-Length SCCB (ELS) facility, a KVM guest can execute
the RSCPI command and determine if the SCCB is large enough to store a
complete reponse. If it is not large enough, then the required length
will be set in the SCCB header.

The caller of the SCLP command is responsible for creating a
large-enough SCCB to store a complete response. Proper checking should
be in place, and the caller should execute the command once-more with
the large-enough SCCB.

This facility also enables an extended SCCB for the Read CPU Info
(RCPUI) command.

When this facility is enabled, the boundary violation response cannot
be a result from the RSCPI, RSCPI Forced, or RCPUI commands.

In order to tolerate kernels that do not yet have full support for this
feature, a "fixed" offset to the start of the CPU Entries within the
Read SCP Info struct is set to allow for the original 248 max entries
when this feature is disabled.

Additionally, this is introduced as a CPU feature to protect the guest
from migrating to a machine that does not support storing an extended
SCCB. This could otherwise hinder the VM from being able to read all
available CPU entries after migration (such as during re-ipl).

Signed-off-by: Collin Walling 
Acked-by: Cornelia Huck 
Reviewed-by: Thomas Huth 
---
 hw/s390x/sclp.c | 43 +
 include/hw/s390x/sclp.h |  1 +
 target/s390x/cpu_features_def.h.inc |  1 +
 target/s390x/gen-features.c |  1 +
 target/s390x/kvm.c  |  8 ++
 5 files changed, 48 insertions(+), 6 deletions(-)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index 1df67c99bf..caf40f41b6 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -49,13 +49,30 @@ static inline bool sclp_command_code_valid(uint32_t code)
 return false;
 }
 
-static bool sccb_verify_boundary(uint64_t sccb_addr, uint16_t sccb_len)
+static bool sccb_verify_boundary(uint64_t sccb_addr, uint16_t sccb_len,
+ uint32_t code)
 {
 uint64_t sccb_max_addr = sccb_addr + sccb_len - 1;
 uint64_t sccb_boundary = (sccb_addr & PAGE_MASK) + PAGE_SIZE;
 
-if (sccb_max_addr < sccb_boundary) {
-return true;
+switch (code & SCLP_CMD_CODE_MASK) {
+case SCLP_CMDW_READ_SCP_INFO:
+case SCLP_CMDW_READ_SCP_INFO_FORCED:
+case SCLP_CMDW_READ_CPU_INFO:
+/*
+ * An extended-length SCCB is only allowed for Read SCP/CPU Info and
+ * is allowed to exceed the 4k boundary. The respective commands will
+ * set the length field to the required length if an insufficient
+ * SCCB length is provided.
+ */
+if (s390_has_feat(S390_FEAT_EXTENDED_LENGTH_SCCB)) {
+return true;
+}
+/* fallthrough */
+default:
+if (sccb_max_addr < sccb_boundary) {
+return true;
+}
 }
 
 return false;
@@ -80,6 +97,12 @@ static void prepare_cpu_entries(MachineState *ms, CPUEntry 
*entry, int *count)
 
 #define SCCB_REQ_LEN(s, max_cpus) (sizeof(s) + max_cpus * sizeof(CPUEntry))
 
+static inline bool ext_len_sccb_supported(SCCBHeader header)
+{
+return s390_has_feat(S390_FEAT_EXTENDED_LENGTH_SCCB) &&
+   header.control_mask[2] & SCLP_VARIABLE_LENGTH_RESPONSE;
+}
+
 /* Provide information about the configuration, CPUs and storage */
 static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 {
@@ -89,10 +112,15 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 int rnsize, rnmax;
 IplParameterBlock *ipib = s390_ipl_get_iplb();
 int required_len = SCCB_REQ_LEN(ReadInfo, machine->possible_cpus->len);
-int offset_cpu = offsetof(ReadInfo, entries);
+int offset_cpu = s390_has_feat(S390_FEAT_EXTENDED_LENGTH_SCCB) ?
+ offsetof(ReadInfo, entries) :
+ SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET;
 CPUEntry *entries_start = (void *)sccb + offset_cpu;
 
 if (be16_to_cpu(sccb->h.length) < required_len) {
+if (ext_len_sccb_supported(sccb->h)) {
+sccb->h.length = cpu_to_be16(required_len);
+}
 sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
 return;
 }
@@ -153,6 +181,9 @@ static void sclp_read_cpu_info(SCLPDevice *sclp, SCCB *sccb)
 int required_len = SCCB_REQ_LEN(ReadCpuInfo, machine->possible_cpus->len);
 
 if (be16_to_cpu(sccb->h.length) < required_len) {
+if (ext_len_sccb_supported(sccb->h)) {
+sccb->h.length = cpu_to_be16(required_len);
+}
 sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
 return;
 }
@@

[PATCH v6 7/8] s390/kvm: header sync for diag318

2020-09-15 Thread Collin Walling

Signed-off-by: Collin Walling 
---
 linux-headers/asm-s390/kvm.h | 7 +--
 linux-headers/linux/kvm.h| 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/linux-headers/asm-s390/kvm.h b/linux-headers/asm-s390/kvm.h
index 0138ccb0d8..f053b8304a 100644
--- a/linux-headers/asm-s390/kvm.h
+++ b/linux-headers/asm-s390/kvm.h
@@ -231,11 +231,13 @@ struct kvm_guest_debug_arch {
 #define KVM_SYNC_GSCB   (1UL << 9)
 #define KVM_SYNC_BPBC   (1UL << 10)
 #define KVM_SYNC_ETOKEN (1UL << 11)
+#define KVM_SYNC_DIAG318 (1UL << 12)
 
 #define KVM_SYNC_S390_VALID_FIELDS \
(KVM_SYNC_PREFIX | KVM_SYNC_GPRS | KVM_SYNC_ACRS | KVM_SYNC_CRS | \
 KVM_SYNC_ARCH0 | KVM_SYNC_PFAULT | KVM_SYNC_VRS | KVM_SYNC_RICCB | \
-KVM_SYNC_FPRS | KVM_SYNC_GSCB | KVM_SYNC_BPBC | KVM_SYNC_ETOKEN)
+KVM_SYNC_FPRS | KVM_SYNC_GSCB | KVM_SYNC_BPBC | KVM_SYNC_ETOKEN | \
+KVM_SYNC_DIAG318)
 
 /* length and alignment of the sdnx as a power of two */
 #define SDNXC 8
@@ -264,7 +266,8 @@ struct kvm_sync_regs {
__u8 reserved2 : 7;
__u8 padding1[51];  /* riccb needs to be 64byte aligned */
__u8 riccb[64]; /* runtime instrumentation controls block */
-   __u8 padding2[192]; /* sdnx needs to be 256byte aligned */
+   __u64 diag318;  /* diagnose 0x318 info */
+   __u8 padding2[184]; /* sdnx needs to be 256byte aligned */
union {
__u8 sdnx[SDNXL];  /* state description annex */
struct {
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index a28c366737..a789bfb5a2 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1031,6 +1031,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SECURE_GUEST 181
 #define KVM_CAP_HALT_POLL 182
 #define KVM_CAP_ASYNC_PF_INT 183
+#define KVM_CAP_S390_DIAG318 184
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
-- 
2.26.2

[PATCH v6 2/8] s390/sclp: rework sclp boundary checks

2020-09-15 Thread Collin Walling

Rework the SCLP boundary check to account for different SCLP commands
(eventually) allowing different boundary sizes.

Signed-off-by: Collin Walling 
Acked-by: Janosch Frank 
Reviewed-by: Cornelia Huck 
---
 hw/s390x/sclp.c | 19 ++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index 28b973de8f..a37cfbf534 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -49,6 +49,18 @@ static inline bool sclp_command_code_valid(uint32_t code)
 return false;
 }
 
+static bool sccb_verify_boundary(uint64_t sccb_addr, uint16_t sccb_len)
+{
+uint64_t sccb_max_addr = sccb_addr + sccb_len - 1;
+uint64_t sccb_boundary = (sccb_addr & PAGE_MASK) + PAGE_SIZE;
+
+if (sccb_max_addr < sccb_boundary) {
+return true;
+}
+
+return false;
+}
+
 static void prepare_cpu_entries(MachineState *ms, CPUEntry *entry, int *count)
 {
 uint8_t features[SCCB_CPU_FEATURE_LEN] = { 0 };
@@ -229,6 +241,11 @@ int sclp_service_call_protected(CPUS390XState *env, 
uint64_t sccb,
 goto out_write;
 }
 
+if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb.h.length))) {
+work_sccb.h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
+goto out_write;
+}
+
 sclp_c->execute(sclp, _sccb, code);
 out_write:
 s390_cpu_pv_mem_write(env_archcpu(env), 0, _sccb,
@@ -274,7 +291,7 @@ int sclp_service_call(CPUS390XState *env, uint64_t sccb, 
uint32_t code)
 goto out_write;
 }
 
-if ((sccb + be16_to_cpu(work_sccb.h.length)) > ((sccb & PAGE_MASK) + 
PAGE_SIZE)) {
+if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb.h.length))) {
 work_sccb.h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
 goto out_write;
 }
-- 
2.26.2

[PATCH v6 4/8] s390/sclp: check sccb len before filling in data

2020-09-15 Thread Collin Walling

The SCCB must be checked for a sufficient length before it is filled
with any data. If the length is insufficient, then the SCLP command
is suppressed and the proper response code is set in the SCCB header.

While we're at it, let's cleanup the length check by placing the
calculation inside a macro.

Fixes: 832be0d8a3bb ("s390x: sclp: Report insufficient SCCB length")
Signed-off-by: Collin Walling 
Reviewed-by: Janosch Frank 
Reviewed-by: David Hildenbrand 
Reviewed-by: Cornelia Huck 
Reviewed-by: Thomas Huth 
---
 hw/s390x/sclp.c | 26 ++
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index 4ae6fb400b..0d54075309 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -78,6 +78,8 @@ static void prepare_cpu_entries(MachineState *ms, CPUEntry 
*entry, int *count)
 }
 }
 
+#define SCCB_REQ_LEN(s, max_cpus) (sizeof(s) + max_cpus * sizeof(CPUEntry))
+
 /* Provide information about the configuration, CPUs and storage */
 static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 {
@@ -86,6 +88,12 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 int cpu_count;
 int rnsize, rnmax;
 IplParameterBlock *ipib = s390_ipl_get_iplb();
+int required_len = SCCB_REQ_LEN(ReadInfo, machine->possible_cpus->len);
+
+if (be16_to_cpu(sccb->h.length) < required_len) {
+sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
+return;
+}
 
 /* CPU information */
 prepare_cpu_entries(machine, read_info->entries, _count);
@@ -95,12 +103,6 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 
 read_info->ibc_val = cpu_to_be32(s390_get_ibc_val());
 
-if (be16_to_cpu(sccb->h.length) <
-(sizeof(ReadInfo) + cpu_count * sizeof(CPUEntry))) {
-sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
-return;
-}
-
 /* Configuration Characteristic (Extension) */
 s390_get_feat_block(S390_FEAT_TYPE_SCLP_CONF_CHAR,
  read_info->conf_char);
@@ -146,18 +148,18 @@ static void sclp_read_cpu_info(SCLPDevice *sclp, SCCB 
*sccb)
 MachineState *machine = MACHINE(qdev_get_machine());
 ReadCpuInfo *cpu_info = (ReadCpuInfo *) sccb;
 int cpu_count;
+int required_len = SCCB_REQ_LEN(ReadCpuInfo, machine->possible_cpus->len);
+
+if (be16_to_cpu(sccb->h.length) < required_len) {
+sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
+return;
+}
 
 prepare_cpu_entries(machine, cpu_info->entries, _count);
 cpu_info->nr_configured = cpu_to_be16(cpu_count);
 cpu_info->offset_configured = cpu_to_be16(offsetof(ReadCpuInfo, entries));
 cpu_info->nr_standby = cpu_to_be16(0);
 
-if (be16_to_cpu(sccb->h.length) <
-(sizeof(ReadCpuInfo) + cpu_count * sizeof(CPUEntry))) {
-sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
-return;
-}
-
 /* The standby offset is 16-byte for each CPU */
 cpu_info->offset_standby = cpu_to_be16(cpu_info->offset_configured
 + cpu_info->nr_configured*sizeof(CPUEntry));
-- 
2.26.2

[PATCH v6 5/8] s390/sclp: use cpu offset to locate cpu entries

2020-09-15 Thread Collin Walling

The start of the CPU entry region in the Read SCP Info response data is
denoted by the offset_cpu field. As such, QEMU needs to begin creating
entries at this address.

This is in preparation for when Read SCP Info inevitably introduces new
bytes that push the start of the CPUEntry field further away.

Read CPU Info is unlikely to ever change, so let's not bother
accounting for the offset there.

Signed-off-by: Collin Walling 
Reviewed-by: Thomas Huth 
Reviewed-by: Cornelia Huck 
---
 hw/s390x/sclp.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index 0d54075309..1df67c99bf 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -89,6 +89,8 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 int rnsize, rnmax;
 IplParameterBlock *ipib = s390_ipl_get_iplb();
 int required_len = SCCB_REQ_LEN(ReadInfo, machine->possible_cpus->len);
+int offset_cpu = offsetof(ReadInfo, entries);
+CPUEntry *entries_start = (void *)sccb + offset_cpu;
 
 if (be16_to_cpu(sccb->h.length) < required_len) {
 sccb->h.response_code = cpu_to_be16(SCLP_RC_INSUFFICIENT_SCCB_LENGTH);
@@ -96,9 +98,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 }
 
 /* CPU information */
-prepare_cpu_entries(machine, read_info->entries, _count);
+prepare_cpu_entries(machine, entries_start, _count);
 read_info->entries_cpu = cpu_to_be16(cpu_count);
-read_info->offset_cpu = cpu_to_be16(offsetof(ReadInfo, entries));
+read_info->offset_cpu = cpu_to_be16(offset_cpu);
 read_info->highest_cpu = cpu_to_be16(machine->smp.max_cpus - 1);
 
 read_info->ibc_val = cpu_to_be32(s390_get_ibc_val());
-- 
2.26.2

[PATCH v6 1/8] s390/sclp: get machine once during read scp/cpu info

2020-09-15 Thread Collin Walling

Functions within read scp/cpu info will need access to the machine
state. Let's make a call to retrieve the machine state once and
pass the appropriate data to the respective functions.

Signed-off-by: Collin Walling 
Reviewed-by: David Hildenbrand 
Reviewed-by: Thomas Huth 
Reviewed-by: Janosch Frank 
Reviewed-by: Cornelia Huck 
---
 hw/s390x/sclp.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index a0ce444b4b..28b973de8f 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -49,9 +49,8 @@ static inline bool sclp_command_code_valid(uint32_t code)
 return false;
 }
 
-static void prepare_cpu_entries(SCLPDevice *sclp, CPUEntry *entry, int *count)
+static void prepare_cpu_entries(MachineState *ms, CPUEntry *entry, int *count)
 {
-MachineState *ms = MACHINE(qdev_get_machine());
 uint8_t features[SCCB_CPU_FEATURE_LEN] = { 0 };
 int i;
 
@@ -77,7 +76,7 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 IplParameterBlock *ipib = s390_ipl_get_iplb();
 
 /* CPU information */
-prepare_cpu_entries(sclp, read_info->entries, _count);
+prepare_cpu_entries(machine, read_info->entries, _count);
 read_info->entries_cpu = cpu_to_be16(cpu_count);
 read_info->offset_cpu = cpu_to_be16(offsetof(ReadInfo, entries));
 read_info->highest_cpu = cpu_to_be16(machine->smp.max_cpus - 1);
@@ -132,10 +131,11 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 /* Provide information about the CPU */
 static void sclp_read_cpu_info(SCLPDevice *sclp, SCCB *sccb)
 {
+MachineState *machine = MACHINE(qdev_get_machine());
 ReadCpuInfo *cpu_info = (ReadCpuInfo *) sccb;
 int cpu_count;
 
-prepare_cpu_entries(sclp, cpu_info->entries, _count);
+prepare_cpu_entries(machine, cpu_info->entries, _count);
 cpu_info->nr_configured = cpu_to_be16(cpu_count);
 cpu_info->offset_configured = cpu_to_be16(offsetof(ReadCpuInfo, entries));
 cpu_info->nr_standby = cpu_to_be16(0);
-- 
2.26.2

[PATCH v6 8/8] s390: guest support for diagnose 0x318

2020-09-15 Thread Collin Walling

DIAGNOSE 0x318 (diag318) is an s390 instruction that allows the storage
of diagnostic information that is collected by the firmware in the case
of hardware/firmware service events.

QEMU handles the instruction by storing the info in the CPU state. A
subsequent register sync will communicate the data to the hypervisor.

QEMU handles the migration via a VM State Description.

This feature depends on the Extended-Length SCCB (els) feature. If
els is not present, then a warning will be printed and the SCLP bit
that allows the Linux kernel to execute the instruction will not be
set.

Availability of this instruction is determined by byte 134 (aka fac134)
bit 0 of the SCLP Read Info block. This coincidentally expands into the
space used for CPU entries, which means VMs running with the diag318
capability may not be able to read information regarding all CPUs
unless the guest kernel supports an extended-length SCCB.

This feature is not supported in protected virtualization mode.

Signed-off-by: Collin Walling 
Acked-by: Janosch Frank 
---
 hw/s390x/sclp.c |  5 
 include/hw/s390x/sclp.h |  8 ++
 target/s390x/cpu.h  |  2 ++
 target/s390x/cpu_features.h |  1 +
 target/s390x/cpu_features_def.h.inc |  3 +++
 target/s390x/cpu_models.c   |  1 +
 target/s390x/gen-features.c |  1 +
 target/s390x/kvm.c  | 39 +
 target/s390x/machine.c  | 17 +
 9 files changed, 77 insertions(+)

diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index caf40f41b6..00f1e4648d 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -139,6 +139,11 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
 s390_get_feat_block(S390_FEAT_TYPE_SCLP_CONF_CHAR_EXT,
  read_info->conf_char_ext);
 
+if (s390_has_feat(S390_FEAT_EXTENDED_LENGTH_SCCB)) {
+s390_get_feat_block(S390_FEAT_TYPE_SCLP_FAC134,
+_info->fac134);
+}
+
 read_info->facilities = cpu_to_be64(SCLP_HAS_CPU_INFO |
 SCLP_HAS_IOA_RECONFIG);
 
diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
index 141e57f765..c84eb8ac65 100644
--- a/include/hw/s390x/sclp.h
+++ b/include/hw/s390x/sclp.h
@@ -133,7 +133,15 @@ typedef struct ReadInfo {
 uint16_t highest_cpu;
 uint8_t  _reserved5[124 - 122]; /* 122-123 */
 uint32_t hmfai;
+uint8_t  _reserved7[134 - 128]; /* 128-133 */
+uint8_t  fac134;
+uint8_t  _reserved8[144 - 135]; /* 135-143 */
 struct CPUEntry entries[];
+/*
+ * When the Extended-Length SCCB (ELS) feature is enabled the
+ * start of the entries field begins at an offset denoted by the
+ * offset_cpu field, otherwise it's at an offset of 128.
+ */
 } QEMU_PACKED ReadInfo;
 
 typedef struct ReadCpuInfo {
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 035427521c..f875ebf0f4 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -112,6 +112,8 @@ struct CPUS390XState {
 uint16_t external_call_addr;
 DECLARE_BITMAP(emergency_signals, S390_MAX_CPUS);
 
+uint64_t diag318_info;
+
 /* Fields up to this point are cleared by a CPU reset */
 struct {} end_reset_fields;
 
diff --git a/target/s390x/cpu_features.h b/target/s390x/cpu_features.h
index 2a29475493..ef52ffce83 100644
--- a/target/s390x/cpu_features.h
+++ b/target/s390x/cpu_features.h
@@ -23,6 +23,7 @@ typedef enum {
 S390_FEAT_TYPE_STFL,
 S390_FEAT_TYPE_SCLP_CONF_CHAR,
 S390_FEAT_TYPE_SCLP_CONF_CHAR_EXT,
+S390_FEAT_TYPE_SCLP_FAC134,
 S390_FEAT_TYPE_SCLP_CPU,
 S390_FEAT_TYPE_MISC,
 S390_FEAT_TYPE_PLO,
diff --git a/target/s390x/cpu_features_def.h.inc 
b/target/s390x/cpu_features_def.h.inc
index 1c04cc18f4..f82b4b5ec1 100644
--- a/target/s390x/cpu_features_def.h.inc
+++ b/target/s390x/cpu_features_def.h.inc
@@ -122,6 +122,9 @@ DEF_FEAT(SIE_CMMA, "cmma", SCLP_CONF_CHAR_EXT, 1, "SIE: 
Collaborative-memory-man
 DEF_FEAT(SIE_PFMFI, "pfmfi", SCLP_CONF_CHAR_EXT, 9, "SIE: PFMF interpretation 
facility")
 DEF_FEAT(SIE_IBS, "ibs", SCLP_CONF_CHAR_EXT, 10, "SIE: 
Interlock-and-broadcast-suppression facility")
 
+/* Features exposed via SCLP SCCB Facilities byte 134 (bit numbers relative to 
byte-134) */
+DEF_FEAT(DIAG_318, "diag318", SCLP_FAC134, 0, "Control program name and 
version codes")
+
 /* Features exposed via SCLP CPU info. */
 DEF_FEAT(SIE_F2, "sief2", SCLP_CPU, 4, "SIE: interception format 2 (Virtual 
SIE)")
 DEF_FEAT(SIE_SKEY, "skey", SCLP_CPU, 5, "SIE: Storage-key facility")
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index c2af226174..9d615f13e7 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -824,6 +824,7 @@ static void check_consistency(const S390CPUModel *model)
 { S390_FEAT_PTFF_STOE, S390_FEAT_MULTIPLE_EPOCH },
 { S390_FEAT_PTFF_STOUE, S390_FEAT_MULTIPLE_EPOCH },
 {

[PATCH v6 0/8] s390: Extended-Length SCCB & DIAGNOSE 0x318

2020-09-15 Thread Collin Walling

Changelog:

v6

• sccb_verify_boundary function:
• s/len/sccb_len
• removed the endian check/conversion of the sccb_len from within 
  this function (caller is now responsible)

• proper endian conversion when using header length to malloc

• use g_autofree for work_sccb

• added r-b's and acks (thanks!)

• added a feature-check fence within the diag_318_handler to ensure
the handler does not complete without proper feature support
• will throw a program exception if handler is invoked but
  feature is not enabled



v5 (comment below pertains to version 5)

Janosch, Thomas, Conny: I've removed your r-b's from patch #3 since I
added some g_mallocs in place and I'd like to make sure things are
done properly there (explained in changelog, but let me know if further
explanation is necessary).

Janosch, please let me know if the changes to #3 are safe under PV.

Thanks.

• removed sccb_verify_length function
- will simply use the length check code that was in place before

• introduced a macro for calculating required SCCB length
- takes a struct and max # of cpus as args

• work_sccb size is now dynamically allocated based on the length
  provided by the guest kernel, instead of always using a static
  4K size
- as such, the SCCB will have to be read twice:
- first time to retrieve the header
- second time with proper size after space for work_sccb 
  is allocated



v4

• added r-b's and ack's (thanks, everyone!)

• renamed boundary and length function

• updated header sync to reflect a change discussed in the respective
KVM patches

• s/data_len/offset_cpu

• added /* fallthrough */ comment in boundary check



v3

• Device IOCTLs removed
- diag 318 info is now communicated via sync_regs

• Reset code removed
- this is now handled in KVM
- diag318_info is stored within the CPU reset portion of the
S390CPUState

• Various cleanups for ELS preliminary patches



v2

• QEMU now handles the instruction call
- as such, the "enable diag 318" IOCTL has been removed

• patch #1 now changes the read scp/cpu info functions to
  retrieve the machine state once
- as such, I have not added any ack's or r-bs since this
  patch differs from the previous version

• patch #3 introduces a new "get_read_scp_info_data_len"
  function in order clean-up the variable data length assignment
  in patch #7
- a comment above this function should help clarify what's
  going on to make things a bit easier to read

• other misc clean ups and fixes
- s/diag318/diag_318 in order to keep the naming scheme
  consistent with Linux and other diag-related code
- s/byte_134/fac134 to align naming scheme with Linux

---

This patch series introduces two features for an s390 KVM quest:
- Extended-Length SCCB (els) for the Read SCP/CPU Info SCLP 
commands
- DIAGNOSE 0x318 (diag_318) enabling / migration handling

The diag 318 feature depends on els and KVM support.

The els feature is handled entirely with QEMU, and does not require 
KVM support.

Both features are made available starting with the zEC12-full model.

These patches are introduced together for two main reasons:
- els allows diag 318 to exist while retaining the original 248 
VCPU max
- diag 318 is presented to show how els is useful

Full els support is dependant on the Linux kernel, which must react
to the SCLP response code and set an appropriate-length SCCB. 

A user should take care when tuning the CPU model for a VM.
If a user defines a VM with els support and specifies 248 CPUs, but
the guest Linux kernel cannot react to the SCLP response code, then
the guest will crash immediately upon kernel startup.

Collin L. Walling (8):
  s390/sclp: get machine once during read scp/cpu info
  s390/sclp: rework sclp boundary checks
  s390/sclp: read sccb from mem based on provided length
  s390/sclp: check sccb len before filling in data
  s390/sclp: use cpu offset to locate cpu entries
  s390/sclp: add extended-length sccb support for kvm guest
  s390/kvm: header sync for diag318
  s390: guest support for diagnose 0x318

 hw/s390x/event-facility.c   |   2 +-
 hw/s390x/sclp.c | 142 
 include/hw/s390x/sclp.h |  11 ++-
 linux-headers/asm-s390/kvm.h|   7 +-
 linux-headers/linux/kvm.h   |   1 +
 target/s390x/cpu.h  |   2 +
 target/s390x/cpu_features.h |   1 +
 target/s390x/cpu_features_def.h.inc |   4 +
 target/s390x/cpu_models.c   |   1 +
 target/s390x/gen-features.c |   2 +
 target/s390x/kvm.c

[PATCH v6 3/8] s390/sclp: read sccb from mem based on provided length

2020-09-15 Thread Collin Walling

The header contained within the SCCB passed to the SCLP service call
contains the actual length of the SCCB. Instead of allocating a static
4K size for the work sccb, let's allow for a variable size determined
by the value in the header. The proper checks are already in place to
ensure the SCCB length is sufficent to store a full response and that
the length does not cross any explicitly-set boundaries.

Signed-off-by: Collin Walling 
---
 hw/s390x/event-facility.c |  2 +-
 hw/s390x/sclp.c   | 55 ++-
 include/hw/s390x/sclp.h   |  2 +-
 3 files changed, 33 insertions(+), 26 deletions(-)

diff --git a/hw/s390x/event-facility.c b/hw/s390x/event-facility.c
index 645b4080c5..ed92ce510d 100644
--- a/hw/s390x/event-facility.c
+++ b/hw/s390x/event-facility.c
@@ -213,7 +213,7 @@ static uint16_t handle_sccb_read_events(SCLPEventFacility 
*ef, SCCB *sccb,
 
 event_buf = >ebh;
 event_buf->length = 0;
-slen = sizeof(sccb->data);
+slen = sccb_data_len(sccb);
 
 rc = SCLP_RC_NO_EVENT_BUFFERS_STORED;
 
diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index a37cfbf534..4ae6fb400b 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -231,25 +231,29 @@ int sclp_service_call_protected(CPUS390XState *env, 
uint64_t sccb,
 {
 SCLPDevice *sclp = get_sclp_device();
 SCLPDeviceClass *sclp_c = SCLP_GET_CLASS(sclp);
-SCCB work_sccb;
-hwaddr sccb_len = sizeof(SCCB);
+SCCBHeader header;
+g_autofree SCCB *work_sccb = NULL;
 
-s390_cpu_pv_mem_read(env_archcpu(env), 0, _sccb, sccb_len);
+s390_cpu_pv_mem_read(env_archcpu(env), 0, , sizeof(SCCBHeader));
+
+work_sccb = g_malloc0(be16_to_cpu(header.length));
+s390_cpu_pv_mem_read(env_archcpu(env), 0, work_sccb,
+ be16_to_cpu(header.length));
 
 if (!sclp_command_code_valid(code)) {
-work_sccb.h.response_code = cpu_to_be16(SCLP_RC_INVALID_SCLP_COMMAND);
+work_sccb->h.response_code = cpu_to_be16(SCLP_RC_INVALID_SCLP_COMMAND);
 goto out_write;
 }
 
-if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb.h.length))) {
-work_sccb.h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
+if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb->h.length))) {
+work_sccb->h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
 goto out_write;
 }
 
-sclp_c->execute(sclp, _sccb, code);
+sclp_c->execute(sclp, work_sccb, code);
 out_write:
-s390_cpu_pv_mem_write(env_archcpu(env), 0, _sccb,
-  be16_to_cpu(work_sccb.h.length));
+s390_cpu_pv_mem_write(env_archcpu(env), 0, work_sccb,
+  be16_to_cpu(work_sccb->h.length));
 sclp_c->service_interrupt(sclp, SCLP_PV_DUMMY_ADDR);
 return 0;
 }
@@ -258,9 +262,8 @@ int sclp_service_call(CPUS390XState *env, uint64_t sccb, 
uint32_t code)
 {
 SCLPDevice *sclp = get_sclp_device();
 SCLPDeviceClass *sclp_c = SCLP_GET_CLASS(sclp);
-SCCB work_sccb;
-
-hwaddr sccb_len = sizeof(SCCB);
+SCCBHeader header;
+g_autofree SCCB *work_sccb = NULL;
 
 /* first some basic checks on program checks */
 if (env->psw.mask & PSW_MASK_PSTATE) {
@@ -274,32 +277,36 @@ int sclp_service_call(CPUS390XState *env, uint64_t sccb, 
uint32_t code)
 return -PGM_SPECIFICATION;
 }
 
+/* the header contains the actual length of the sccb */
+cpu_physical_memory_read(sccb, , sizeof(SCCBHeader));
+
+/* Valid sccb sizes */
+if (be16_to_cpu(header.length) < sizeof(SCCBHeader)) {
+return -PGM_SPECIFICATION;
+}
+
 /*
  * we want to work on a private copy of the sccb, to prevent guests
  * from playing dirty tricks by modifying the memory content after
  * the host has checked the values
  */
-cpu_physical_memory_read(sccb, _sccb, sccb_len);
-
-/* Valid sccb sizes */
-if (be16_to_cpu(work_sccb.h.length) < sizeof(SCCBHeader)) {
-return -PGM_SPECIFICATION;
-}
+work_sccb = g_malloc0(be16_to_cpu(header.length));
+cpu_physical_memory_read(sccb, work_sccb, be16_to_cpu(header.length));
 
 if (!sclp_command_code_valid(code)) {
-work_sccb.h.response_code = cpu_to_be16(SCLP_RC_INVALID_SCLP_COMMAND);
+work_sccb->h.response_code = cpu_to_be16(SCLP_RC_INVALID_SCLP_COMMAND);
 goto out_write;
 }
 
-if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb.h.length))) {
-work_sccb.h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
+if (!sccb_verify_boundary(sccb, be16_to_cpu(work_sccb->h.length))) {
+work_sccb->h.response_code = 
cpu_to_be16(SCLP_RC_SCCB_BOUNDARY_VIOLATION);
 goto out_write;
 }
 
-sclp_c->execute(sclp, _sccb, code);
+sclp_c->execute(sclp, work_sccb, code);
 out_write:
-cpu_physical_memory_write(sccb, _sccb,
-  be16_to_cpu(work_sccb.h.length));
+cpu_physical_memory_write(sccb, work_sccb,
+

[PATCH v3 4/5] s390x/pci: Add routine to get the vfio dma available count

2020-09-15 Thread Matthew Rosato

Create new files for separating out vfio-specific work for s390
pci. Add the first such routine, which issues VFIO_IOMMU_GET_INFO
ioctl to collect the current dma available count.

Signed-off-by: Matthew Rosato 
---
 hw/s390x/meson.build |  1 +
 hw/s390x/s390-pci-vfio.c | 54 
 hw/s390x/s390-pci-vfio.h | 17 +++
 3 files changed, 72 insertions(+)
 create mode 100644 hw/s390x/s390-pci-vfio.c
 create mode 100644 hw/s390x/s390-pci-vfio.h

diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
index b63782d..ed2f66b 100644
--- a/hw/s390x/meson.build
+++ b/hw/s390x/meson.build
@@ -10,6 +10,7 @@ s390x_ss.add(files(
   's390-ccw.c',
   's390-pci-bus.c',
   's390-pci-inst.c',
+  's390-pci-vfio.c',
   's390-skeys.c',
   's390-stattrib.c',
   's390-virtio-hcall.c',
diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c
new file mode 100644
index 000..75e3ac1
--- /dev/null
+++ b/hw/s390x/s390-pci-vfio.c
@@ -0,0 +1,54 @@
+/*
+ * s390 vfio-pci interfaces
+ *
+ * Copyright 2020 IBM Corp.
+ * Author(s): Matthew Rosato 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include 
+
+#include "qemu/osdep.h"
+#include "s390-pci-vfio.h"
+#include "hw/vfio/vfio-common.h"
+
+/*
+ * Get the current DMA available count from vfio.  Returns true if vfio is
+ * limiting DMA requests, false otherwise.  The current available count read
+ * from vfio is returned in avail.
+ */
+bool s390_pci_update_dma_avail(int fd, unsigned int *avail)
+{
+g_autofree struct vfio_iommu_type1_info *info;
+uint32_t argsz;
+int ret;
+
+assert(avail);
+
+argsz = sizeof(struct vfio_iommu_type1_info);
+info = g_malloc0(argsz);
+info->argsz = argsz;
+/*
+ * If the specified argsz is not large enough to contain all
+ * capabilities it will be updated upon return.  In this case
+ * use the updated value to get the entire capability chain.
+ */
+ret = ioctl(fd, VFIO_IOMMU_GET_INFO, info);
+if (argsz != info->argsz) {
+argsz = info->argsz;
+info = g_realloc(info, argsz);
+info->argsz = argsz;
+ret = ioctl(fd, VFIO_IOMMU_GET_INFO, info);
+}
+
+if (ret) {
+return false;
+}
+
+/* If the capability exists, update with the current value */
+return vfio_get_info_dma_avail(info, avail);
+}
+
diff --git a/hw/s390x/s390-pci-vfio.h b/hw/s390x/s390-pci-vfio.h
new file mode 100644
index 000..2a5a261
--- /dev/null
+++ b/hw/s390x/s390-pci-vfio.h
@@ -0,0 +1,17 @@
+/*
+ * s390 vfio-pci interfaces
+ *
+ * Copyright 2020 IBM Corp.
+ * Author(s): Matthew Rosato 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#ifndef HW_S390_PCI_VFIO_H
+#define HW_S390_PCI_VFIO_H
+
+bool s390_pci_update_dma_avail(int fd, unsigned int *avail);
+
+#endif
-- 
1.8.3.1

[PATCH v3 2/5] vfio: Create shared routine for scanning info capabilities

2020-09-15 Thread Matthew Rosato

Rather than duplicating the same loop in multiple locations,
create a static function to do the work.

Signed-off-by: Matthew Rosato 
---
 hw/vfio/common.c | 21 +
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index 3335714..eba7b55 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -825,17 +825,12 @@ static void vfio_listener_release(VFIOContainer 
*container)
 }
 }
 
-struct vfio_info_cap_header *
-vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
+static struct vfio_info_cap_header *
+vfio_get_cap(void *ptr, uint32_t cap_offset, uint16_t id)
 {
 struct vfio_info_cap_header *hdr;
-void *ptr = info;
-
-if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) {
-return NULL;
-}
 
-for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) {
+for (hdr = ptr + cap_offset; hdr != ptr; hdr = ptr + hdr->next) {
 if (hdr->id == id) {
 return hdr;
 }
@@ -844,6 +839,16 @@ vfio_get_region_info_cap(struct vfio_region_info *info, 
uint16_t id)
 return NULL;
 }
 
+struct vfio_info_cap_header *
+vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
+{
+if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) {
+return NULL;
+}
+
+return vfio_get_cap((void *)info, info->cap_offset, id);
+}
+
 static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
   struct vfio_region_info *info)
 {
-- 
1.8.3.1

[PATCH v3 1/5] linux-headers: update against 5.9-rc5

2020-09-15 Thread Matthew Rosato

PLACEHOLDER as the kernel patch driving the need for this ("vfio iommu: Add
dma available capability") is not merged yet.

Signed-off-by: Matthew Rosato 
---
 include/standard-headers/drm/drm_fourcc.h  | 140 +
 include/standard-headers/linux/ethtool.h   |  87 +
 include/standard-headers/linux/input-event-codes.h |   3 +-
 include/standard-headers/linux/vhost_types.h   |  11 ++
 include/standard-headers/linux/virtio_9p.h |   4 +-
 include/standard-headers/linux/virtio_blk.h|  26 ++--
 include/standard-headers/linux/virtio_config.h |   8 +-
 include/standard-headers/linux/virtio_console.h|   8 +-
 include/standard-headers/linux/virtio_net.h|   6 +-
 include/standard-headers/linux/virtio_scsi.h   |  20 +--
 linux-headers/asm-generic/unistd.h |   6 +-
 linux-headers/asm-mips/unistd_n32.h|   1 +
 linux-headers/asm-mips/unistd_n64.h|   1 +
 linux-headers/asm-mips/unistd_o32.h|   1 +
 linux-headers/asm-powerpc/kvm.h|   5 +
 linux-headers/asm-powerpc/unistd_32.h  |   1 +
 linux-headers/asm-powerpc/unistd_64.h  |   1 +
 linux-headers/asm-s390/kvm.h   |   7 +-
 linux-headers/asm-s390/unistd_32.h |   1 +
 linux-headers/asm-s390/unistd_64.h |   1 +
 linux-headers/asm-x86/unistd_32.h  |   1 +
 linux-headers/asm-x86/unistd_64.h  |   1 +
 linux-headers/asm-x86/unistd_x32.h |   1 +
 linux-headers/linux/kvm.h  |  10 +-
 linux-headers/linux/vfio.h |  18 ++-
 linux-headers/linux/vhost.h|   2 +
 26 files changed, 328 insertions(+), 43 deletions(-)

diff --git a/include/standard-headers/drm/drm_fourcc.h 
b/include/standard-headers/drm/drm_fourcc.h
index 909a667..0de1a55 100644
--- a/include/standard-headers/drm/drm_fourcc.h
+++ b/include/standard-headers/drm/drm_fourcc.h
@@ -235,6 +235,12 @@ extern "C" {
 #define DRM_FORMAT_NV61fourcc_code('N', 'V', '6', '1') /* 2x1 
subsampled Cb:Cr plane */
 #define DRM_FORMAT_NV24fourcc_code('N', 'V', '2', '4') /* 
non-subsampled Cr:Cb plane */
 #define DRM_FORMAT_NV42fourcc_code('N', 'V', '4', '2') /* 
non-subsampled Cb:Cr plane */
+/*
+ * 2 plane YCbCr
+ * index 0 = Y plane, [39:0] Y3:Y2:Y1:Y0 little endian
+ * index 1 = Cr:Cb plane, [39:0] Cr1:Cb1:Cr0:Cb0 little endian
+ */
+#define DRM_FORMAT_NV15fourcc_code('N', 'V', '1', '5') /* 2x2 
subsampled Cr:Cb plane */
 
 /*
  * 2 plane YCbCr MSB aligned
@@ -264,6 +270,22 @@ extern "C" {
  */
 #define DRM_FORMAT_P016fourcc_code('P', '0', '1', '6') /* 2x2 
subsampled Cr:Cb plane 16 bits per channel */
 
+/* 3 plane non-subsampled (444) YCbCr
+ * 16 bits per component, but only 10 bits are used and 6 bits are padded
+ * index 0: Y plane, [15:0] Y:x [10:6] little endian
+ * index 1: Cb plane, [15:0] Cb:x [10:6] little endian
+ * index 2: Cr plane, [15:0] Cr:x [10:6] little endian
+ */
+#define DRM_FORMAT_Q410fourcc_code('Q', '4', '1', '0')
+
+/* 3 plane non-subsampled (444) YCrCb
+ * 16 bits per component, but only 10 bits are used and 6 bits are padded
+ * index 0: Y plane, [15:0] Y:x [10:6] little endian
+ * index 1: Cr plane, [15:0] Cr:x [10:6] little endian
+ * index 2: Cb plane, [15:0] Cb:x [10:6] little endian
+ */
+#define DRM_FORMAT_Q401fourcc_code('Q', '4', '0', '1')
+
 /*
  * 3 plane YCbCr
  * index 0: Y plane, [7:0] Y
@@ -308,6 +330,7 @@ extern "C" {
 #define DRM_FORMAT_MOD_VENDOR_BROADCOM 0x07
 #define DRM_FORMAT_MOD_VENDOR_ARM 0x08
 #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09
+#define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a
 
 /* add more to the end as needed */
 
@@ -322,8 +345,33 @@ extern "C" {
  * When adding a new token please document the layout with a code comment,
  * similar to the fourcc codes above. drm_fourcc.h is considered the
  * authoritative source for all of these.
+ *
+ * Generic modifier names:
+ *
+ * DRM_FORMAT_MOD_GENERIC_* definitions are used to provide vendor-neutral 
names
+ * for layouts which are common across multiple vendors. To preserve
+ * compatibility, in cases where a vendor-specific definition already exists 
and
+ * a generic name for it is desired, the common name is a purely symbolic alias
+ * and must use the same numerical value as the original definition.
+ *
+ * Note that generic names should only be used for modifiers which describe
+ * generic layouts (such as pixel re-ordering), which may have
+ * independently-developed support across multiple vendors.
+ *
+ * In future cases where a generic layout is identified before merging with a
+ * vendor-specific modifier, a new 'GENERIC' vendor or modifier using vendor
+ * 'NONE' could be considered. This should only be for obvious, exceptional
+ * cases to avoid polluting

Re: [PATCH] guest agent: Fixes for windows guest agent building on msys2/mingw

2020-09-15 Thread Yonggang Luo

On Wed, Sep 16, 2020 at 1:55 AM Philippe Mathieu-Daudé 
wrote:
>
> On 9/15/20 7:41 PM, 罗勇刚(Yonggang Luo) wrote:
> > On Wed, Sep 16, 2020 at 12:41 AM Philippe Mathieu-Daudé
> > mailto:phi...@redhat.com>> wrote:
> >>
> >> On 9/15/20 6:33 PM, Yonggang Luo wrote:
> >> > error message:
> >> > "cc" "-Iqga/qemu-ga.exe.p" "-Iqga" "-I../qga" "-I." "-Iqapi"
> > "-Itrace" "-Iui" "-Iui/shader" "-IC:/CI-Tools/msys64/mingw64/include"
> > "-IC:/CI-Tools/msys64/mingw64/include/glib-2.0"
> > "-IC:/CI-Tools/msys64/mingw64/lib/glib-2.0/include"
> > "-fdiagnostics-color=auto" "-pipe" "-Wall" "-Winvalid-pch" "-Werror"
> > "-std=gnu99" "-g" "-m64" "-mcx16" "-D_GNU_SOURCE"
> > "-D_FILE_OFFSET_BITS=64" "-D_LARGEFILE_SOURCE" "-Wstrict-prototypes"
> > "-Wredundant-decls" "-Wundef" "-Wwrite-strings" "-Wmissing-prototypes"
> > "-fno-strict-aliasing" "-fno-common" "-fwrapv" "-Wold-style-declaration"
> > "-Wold-style-definition" "-Wtype-limits" "-Wformat-security"
> > "-Wformat-y2k" "-Winit-self" "-Wignored-qualifiers" "-Wempty-body"
> > "-Wnested-externs" "-Wendif-labels" "-Wexpansion-to-defined"
> > "-Wno-missing-include-dirs" "-Wno-shift-negative-value" "-Wno-psabi"
> > "-fstack-protector-strong" "-iquote" "/c/work/xemu/qemu/tcg/i386"
> > "-iquote" "." "-iquote" "/c/work/xemu/qemu" "-iquote"
> > "/c/work/xemu/qemu/accel/tcg" "-iquote" "/c/work/xemu/qemu/include"
> > "-iquote" "/c/work/xemu/qemu/disas/libvixl" "-pthread" "-mms-bitfields"
> > -MD -MQ qga/qemu-ga.exe.p/commands-win32.c.obj -MF
> > "qga/qemu-ga.exe.p/commands-win32.c.obj.d" -o
> > qga/qemu-ga.exe.p/commands-win32.c.obj "-c" ../qga/commands-win32.c -MP
> >> > ../qga/commands-win32.c:62:24: error: redundant redeclaration of
> > 'CM_Get_DevNode_PropertyW' [-Werror=redundant-decls]
> >> >62 | CMAPI CONFIGRET WINAPI CM_Get_DevNode_PropertyW(
> >> >   |^~~~
> >> > In file included from ../qga/commands-win32.c:26:
> >> >
> > C:/CI-Tools/msys64/mingw64/x86_64-w64-mingw32/include/cfgmgr32.h:840:26:
> > note: previous declaration of 'CM_Get_DevNode_PropertyW' was here
> >> >   840 |   CMAPI CONFIGRET WINAPI CM_Get_DevNode_PropertyW(DEVINST
> > dnDevInst, const DEVPROPKEY *PropertyKey, DEVPROPTYPE *PropertyType,
> > PBYTE PropertyBuffer, PULONG PropertyBufferSize, ULONG ulFlags);
> >> >   |  ^~~~
> >> > cc1.exe: all warnings being treated as errors
> >> > make: [Makefile.ninja:5143：qga/qemu-ga.exe.p/commands-win32.c.obj]
> > 错误 1 (已忽略）
> >> >
> >> > This error comes from qemu configure didn't add predefined macro
> > -DUNICODE -D_UNICODE in QEMU_CFLAGS,
> >> > and these too macro are standard config for win32 if using windows
> > wide api.
> >> >
> >> > in cfgmgr32.h
> >> >   CMAPI CONFIGRET WINAPI CM_Get_DevNode_PropertyW(DEVINST dnDevInst,
> > const DEVPROPKEY *PropertyKey, DEVPROPTYPE *PropertyType, PBYTE
> > PropertyBuffer, PULONG PropertyBufferSize, ULONG ulFlags);
> >> > #ifdef UNICODE
> >> > #define CM_Get_DevNode_Property CM_Get_DevNode_PropertyW
> >> > #endif
> >> >
> >> > macro CM_Get_DevNode_Property defined only when UNICODE are defined.
> >> > and in win32, UNICODE and _UNICODE should be defined at the same time
> >> >
> >> > #endif
> >> >
> >> > Signed-off-by: Yonggang Luo  > >
> >> > ---
> >> >  configure | 1 +
> >> >  1 file changed, 1 insertion(+)
> >> >
> >> > diff --git a/configure b/configure
> >> > index 9f141891ea..f5d661af4a 100755
> >> > --- a/configure
> >> > +++ b/configure
> >> > @@ -812,6 +812,7 @@ MINGW32*)
> >> >mingw32="yes"
> >> >hax="yes"
> >> >whpx=""
> >> > +  QEMU_CFLAGS="-DUNICODE -D_UNICODE $QEMU_CFLAGS"
> >>
> >> What about declaring them in include/qemu/osdep.h
> >> instead?
> > That's a good idea.
> > But more compiling error warns me that the windows version of qemu are
> > not in a good situation.
>
> :)
>
> > On Win32, all API should call the wide version so that support for
> > Unicode properly.
> > But obviously the currently qemu didn't do that
> >
> > ```
> [...]
> > ../util/qemu-sockets.c: In function 'socket_sockaddr_to_address_inet':
> > C:/work/xemu/qemu/include/qapi/error.h:319:25: error: format '%s'
> > expects argument of type 'char *', but argument 6 has type 'WCHAR *'
> > {aka 'short unsigned int *'} [-Werror=format=]
> >   319 | (fmt), ## __VA_ARGS__)
> >   | ^
> > ../util/qemu-sockets.c:1235:9: note: in expansion of macro 'error_setg'
> >  1235 | error_setg(errp, "Cannot format numeric socket address:
%s",
> >   | ^~
> > In file included from ../io/dns-resolver.c:26:
> > ../io/dns-resolver.c: In function 'qio_dns_resolver_lookup_sync_inet':
> > C:/work/xemu/qemu/include/qapi/error.h:319:25: error: format '%s'
> > expects argument of type 'char *', but argument 8 has type 'WCHAR *'
> > {aka 'short unsigned int *'} [-Werror=format=]
> >   319 | (fmt), ## __VA_ARGS__)
> >   |

[PATCH v3 0/5] s390x/pci: Accomodate vfio DMA limiting

2020-09-15 Thread Matthew Rosato

Kernel commit 492855939bdb added a limit to the number of outstanding DMA
requests for a type1 vfio container.  However, lazy unmapping in s390 can 
in fact cause quite a large number of outstanding DMA requests to build up
prior to being purged, potentially the entire guest DMA space.  This
results in unexpected 'VFIO_MAP_DMA failed: No space left on device'
conditions seen in QEMU.

This patchset adds support to qemu to retrieve the number of allowable DMA
requests via the VFIO_IOMMU_GET_INFO ioctl.  The patches are separated into
vfio hits which add support for reading in VFIO_IOMMU_GET_INFO capability
chains and getting the per-container dma_avail value, and s390 hits to 
track DMA usage on a per-container basis.

Associated kernel patch:
https://marc.info/?l=kvm=160019703922812=2

Changes from v2:
- Patch 1 (new): Added a placeholder linux-headers sync
- Patch 2: Re-arranged so that this patch that adds a shared routine for
  finding info capabilities is first
- Patch 3: Adjusted vfio_get_info_dma_avail() logic to be able to return
  true when the capability exists but the caller did not provide a buffer
  on input.
- Patch 4 (new): Introduce hw/s390x/s390-pci-vfio.* to better-separate PCI
  emulation code from code that interfaces with vfio
- Patch 4: s/s390_sync_dma_avail/s390_pci_update_dma_avail/ - Since it's
  now moved into a different file it can't be static so I added '_pci'.
- Patch 4: Use g_autofree as suggested - drop the goto/out as a result
- Patch 4+5: Added asserts() in a few locations per suggestion
- Patch 5: lowercase inline function names
- Patch 5: fix dma_avail initialization in rpcit_service_call() and ensure
  we still allow unmaps to call s390_pci_update_iotlb when dma_avail == 0

Matthew Rosato (5):
  linux-headers: update against 5.9-rc5
  vfio: Create shared routine for scanning info capabilities
  vfio: Find DMA available capability
  s390x/pci: Add routine to get the vfio dma available count
  s390x/pci: Honor DMA limits set by vfio

 hw/s390x/meson.build   |   1 +
 hw/s390x/s390-pci-bus.c|  56 -
 hw/s390x/s390-pci-bus.h|   9 ++
 hw/s390x/s390-pci-inst.c   |  34 -
 hw/s390x/s390-pci-inst.h   |   3 +
 hw/s390x/s390-pci-vfio.c   |  54 
 hw/s390x/s390-pci-vfio.h   |  17 +++
 hw/vfio/common.c   |  52 ++--
 include/hw/vfio/vfio-common.h  |   2 +
 include/standard-headers/drm/drm_fourcc.h  | 140 +
 include/standard-headers/linux/ethtool.h   |  87 +
 include/standard-headers/linux/input-event-codes.h |   3 +-
 include/standard-headers/linux/vhost_types.h   |  11 ++
 include/standard-headers/linux/virtio_9p.h |   4 +-
 include/standard-headers/linux/virtio_blk.h|  26 ++--
 include/standard-headers/linux/virtio_config.h |   8 +-
 include/standard-headers/linux/virtio_console.h|   8 +-
 include/standard-headers/linux/virtio_net.h|   6 +-
 include/standard-headers/linux/virtio_scsi.h   |  20 +--
 linux-headers/asm-generic/unistd.h |   6 +-
 linux-headers/asm-mips/unistd_n32.h|   1 +
 linux-headers/asm-mips/unistd_n64.h|   1 +
 linux-headers/asm-mips/unistd_o32.h|   1 +
 linux-headers/asm-powerpc/kvm.h|   5 +
 linux-headers/asm-powerpc/unistd_32.h  |   1 +
 linux-headers/asm-powerpc/unistd_64.h  |   1 +
 linux-headers/asm-s390/kvm.h   |   7 +-
 linux-headers/asm-s390/unistd_32.h |   1 +
 linux-headers/asm-s390/unistd_64.h |   1 +
 linux-headers/asm-x86/unistd_32.h  |   1 +
 linux-headers/asm-x86/unistd_64.h  |   1 +
 linux-headers/asm-x86/unistd_x32.h |   1 +
 linux-headers/linux/kvm.h  |  10 +-
 linux-headers/linux/vfio.h |  18 ++-
 linux-headers/linux/vhost.h|   2 +
 35 files changed, 537 insertions(+), 62 deletions(-)
 create mode 100644 hw/s390x/s390-pci-vfio.c
 create mode 100644 hw/s390x/s390-pci-vfio.h

-- 
1.8.3.1

[PATCH v3 3/5] vfio: Find DMA available capability

2020-09-15 Thread Matthew Rosato

The underlying host may be limiting the number of outstanding DMA
requests for type 1 IOMMU.  Add helper functions to check for the
DMA available capability and retrieve the current number of DMA
mappings allowed.

Signed-off-by: Matthew Rosato 
---
 hw/vfio/common.c  | 31 +++
 include/hw/vfio/vfio-common.h |  2 ++
 2 files changed, 33 insertions(+)

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index eba7b55..50779f9 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -849,6 +849,37 @@ vfio_get_region_info_cap(struct vfio_region_info *info, 
uint16_t id)
 return vfio_get_cap((void *)info, info->cap_offset, id);
 }
 
+static struct vfio_info_cap_header *
+vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t id)
+{
+if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) {
+return NULL;
+}
+
+return vfio_get_cap((void *)info, info->cap_offset, id);
+}
+
+bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
+ unsigned int *avail)
+{
+struct vfio_info_cap_header *hdr;
+struct vfio_iommu_type1_info_dma_avail *cap;
+
+/* If the capability cannot be found, assume no DMA limiting */
+hdr = vfio_get_iommu_type1_info_cap(info,
+VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL);
+if (hdr == NULL) {
+return false;
+}
+
+if (avail != NULL) {
+cap = (void *) hdr;
+*avail = cap->avail;
+}
+
+return true;
+}
+
 static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
   struct vfio_region_info *info)
 {
diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
index c78f3ff..661a380 100644
--- a/include/hw/vfio/vfio-common.h
+++ b/include/hw/vfio/vfio-common.h
@@ -178,6 +178,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, 
Error **errp);
 void vfio_put_group(VFIOGroup *group);
 int vfio_get_device(VFIOGroup *group, const char *name,
 VFIODevice *vbasedev, Error **errp);
+bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
+ unsigned int *avail);
 
 extern const MemoryRegionOps vfio_region_ops;
 typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;
-- 
1.8.3.1

1 2 3 4 5 >

1 - 100 of 472 matches

Mail list logo