Re: [PATCH 05/38] target/riscv: 8-bit Addition & Subtraction Instruction

[Palmer Dabbelt]

On Tue, 25 May 2021 22:43:27 PDT (-0700), zhiwei_...@c-sky.com wrote:


On 5/24/21 9:00 AM, Palmer Dabbelt wrote:

On Mon, 15 Mar 2021 14:22:58 PDT (-0700), alistai...@gmail.com wrote:

On Fri, Feb 12, 2021 at 10:14 AM LIU Zhiwei 
wrote:


Signed-off-by: LIU Zhiwei 


Acked-by: Alistair Francis 


I saw some reviews on the other ones, but since others (like this)
just have acks and haven't had any other traffic I'm going to start here.

It looks like the latest spec is 0.9.4, but the changelog is pretty
minimal between 0.9.5 and 0.9.2:

[0.9.2 -> 0.9.3]

* Changed Zp64 name to Zpsfoperand.
* Added Zprvsfextra for RV64 only instructions.
* Removed SWAP16 encoding. It is an alias of PKBT16.
* Fixed few typos and enhanced precision descriptions on imtermediate
results.

[0.9.3 -> 0.9.4]

* Fixed few typos and enhanced precision descriptions on imtermediate
results.
* Fixed/Changed data types for some intrinsic functions.
* Removed "RV32 Only" for Zpsfoperand.

So I'm just going to stick with reviewing based on the latest spec

and try to keep those differences in mind, assuming we're just
tracking the latest draft here.


Hi Palmer,

It's a good news.

I plan to rebase the patch set and update to the latest specification.

Probably before next week, we can get a v2 patch set.


Sounds good.  I'll keep slowly going through these until the v2 shows up 
and then jump over there.




Zhiwei


Alistair


---
 target/riscv/helper.h   |  9 +++
 target/riscv/insn32.decode  | 11 
 target/riscv/insn_trans/trans_rvp.c.inc | 79 +
 target/riscv/packed_helper.c    | 73 +++
 4 files changed, 172 insertions(+)

diff --git a/target/riscv/helper.h b/target/riscv/helper.h
index 6d622c732a..a69a6b4e84 100644
--- a/target/riscv/helper.h
+++ b/target/riscv/helper.h
@@ -1175,3 +1175,12 @@ DEF_HELPER_3(rstsa16, tl, env, tl, tl)
 DEF_HELPER_3(urstsa16, tl, env, tl, tl)
 DEF_HELPER_3(kstsa16, tl, env, tl, tl)
 DEF_HELPER_3(ukstsa16, tl, env, tl, tl)
+
+DEF_HELPER_3(radd8, tl, env, tl, tl)
+DEF_HELPER_3(uradd8, tl, env, tl, tl)
+DEF_HELPER_3(kadd8, tl, env, tl, tl)
+DEF_HELPER_3(ukadd8, tl, env, tl, tl)
+DEF_HELPER_3(rsub8, tl, env, tl, tl)
+DEF_HELPER_3(ursub8, tl, env, tl, tl)
+DEF_HELPER_3(ksub8, tl, env, tl, tl)
+DEF_HELPER_3(uksub8, tl, env, tl, tl)
diff --git a/target/riscv/insn32.decode b/target/riscv/insn32.decode
index 8815e90476..358dd1fa10 100644
--- a/target/riscv/insn32.decode
+++ b/target/riscv/insn32.decode
@@ -624,3 +624,14 @@ rstsa16    1011011  . . 010 .
111 @r
 urstsa16   1101011  . . 010 . 111 @r
 kstsa16    1100011  . . 010 . 111 @r
 ukstsa16   1110011  . . 010 . 111 @r
+
+add8   0100100  . . 000 . 111 @r
+radd8  100  . . 000 . 111 @r
+uradd8 0010100  . . 000 . 111 @r
+kadd8  0001100  . . 000 . 111 @r
+ukadd8 0011100  . . 000 . 111 @r
+sub8   0100101  . . 000 . 111 @r
+rsub8  101  . . 000 . 111 @r
+ursub8 0010101  . . 000 . 111 @r
+ksub8  0001101  . . 000 . 111 @r
+uksub8 0011101  . . 000 . 111 @r
diff --git a/target/riscv/insn_trans/trans_rvp.c.inc
b/target/riscv/insn_trans/trans_rvp.c.inc
index 0885a4fd45..109f560ec9 100644
--- a/target/riscv/insn_trans/trans_rvp.c.inc
+++ b/target/riscv/insn_trans/trans_rvp.c.inc
@@ -159,3 +159,82 @@ GEN_RVP_R_OOL(rstsa16);
 GEN_RVP_R_OOL(urstsa16);
 GEN_RVP_R_OOL(kstsa16);
 GEN_RVP_R_OOL(ukstsa16);
+
+/* 8-bit Addition & Subtraction Instructions */
+/*
+ *  Copied from tcg-op-gvec.c.
+ *
+ *  Perform a vector addition using normal addition and a mask. 
The mask
+ *  should be the sign bit of each lane.  This 6-operation form is
more
+ *  efficient than separate additions when there are 4 or more
lanes in
+ *  the 64-bit operation.
+ */
+
+static void gen_simd_add_mask(TCGv d, TCGv a, TCGv b, TCGv m)
+{
+    TCGv t1 = tcg_temp_new();
+    TCGv t2 = tcg_temp_new();
+    TCGv t3 = tcg_temp_new();
+
+    tcg_gen_andc_tl(t1, a, m);
+    tcg_gen_andc_tl(t2, b, m);
+    tcg_gen_xor_tl(t3, a, b);
+    tcg_gen_add_tl(d, t1, t2);
+    tcg_gen_and_tl(t3, t3, m);
+    tcg_gen_xor_tl(d, d, t3);
+
+    tcg_temp_free(t1);
+    tcg_temp_free(t2);
+    tcg_temp_free(t3);
+}
+
+static void tcg_gen_simd_add8(TCGv d, TCGv a, TCGv b)
+{
+    TCGv m = tcg_const_tl((target_ulong)dup_const(MO_8, 0x80));
+    gen_simd_add_mask(d, a, b, m);
+    tcg_temp_free(m);
+}
+
+GEN_RVP_R_INLINE(add8, add, 0, trans_add);
+
+/*
+ *  Copied from tcg-op-gvec.c.
+ *
+ *  Perform a vector subtraction using normal subtraction and a mask.
+ *  Compare gen_addv_mask above.
+ */
+static void gen_simd_sub_mask(TCGv d, TCGv a, TCGv b, TCGv m)
+{
+    TCGv t1 = tcg_temp_ne

Re: [PATCH v6 03/13] virtio-gpu: Add udmabuf helpers

[Gerd Hoffmann]

  Hi,

> +#ifdef CONFIG_LINUX

Seems wee need "#if defined(CONFIG_LINUX) && defined(F_GET_SEALS)" here
to cover old linux versions.  I see some build failures in gitlab CI due to
F_GET_SEALS not being defined (crypto-old-nettle for example).

> +#include 

gitlab ci (build-system-alpine):

/usr/include/sys/fcntl.h:1:2: error: #warning redirecting incorrect #include 
 to  [-Werror=cpp]
1 | #warning redirecting incorrect #include  to 

Otherwise series looks good and survived basic testing.

take care,
  Gerd

Re: [PATCH] gitignore: Update with some filetypes

[Viresh Kumar]

On 24-03-21, 16:33, Viresh Kumar wrote:
> Update .gitignore to ignore .swp and .patch files.
> 
> Signed-off-by: Viresh Kumar 
> ---
>  .gitignore | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/.gitignore b/.gitignore
> index 75a4be07240f..eb2553026c5e 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -13,3 +13,5 @@ GTAGS
>  *~
>  *.ast_raw
>  *.depend_raw
> +*.swp
> +*.patch

Ping.

-- 
viresh

[Bug 1878067] Re: Assertion failure in eth_get_gso_type through the e1000e

[Thomas Huth]

Ok, thanks for checking! So let's close this ticket now.

** Changed in: qemu
   Status: Incomplete => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1878067

Title:
  Assertion failure in eth_get_gso_type through the e1000e

Status in QEMU:
  Fix Released

Bug description:
  Hello,
  While fuzzing, I found an input that triggers an assertion failure in
  eth_get_gso_type through the e1000e:

  #1  0x7685755b in __GI_abort () at abort.c:79
  #2  0x77c75dc3 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
  #3  0x77cd0b0a in g_assertion_message_expr () at 
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
  #4  0x56875f33 in eth_get_gso_type (l3_proto=, 
l3_hdr=, l4proto=) at 
/home/alxndr/Development/qemu/net/eth.c:76
  #5  0x565e09ac in net_tx_pkt_get_gso_type (pkt=0x63114800, 
tso_enable=0x1) at /home/alxndr/Development/qemu/hw/net/net_tx_pkt.c:300
  #6  0x565e09ac in net_tx_pkt_build_vheader (pkt=0x63114800, 
tso_enable=, csum_enable=, gso_size=) at /home/alxndr/Development/qemu/hw/net/net_tx_pkt.c:316
  #7  0x5660bdb1 in e1000e_setup_tx_offloads (core=0x7fffeeb754e0, 
tx=0x7fffeeb95748) at /home/alxndr/Development/qemu/hw/net/e1000e_core.c:637
  #8  0x5660bdb1 in e1000e_tx_pkt_send (core=0x7fffeeb754e0, 
tx=0x7fffeeb95748, queue_index=) at 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:658
  #9  0x5660bdb1 in e1000e_process_tx_desc (core=0x7fffeeb754e0, 
tx=0x7fffeeb95748, dp=, queue_index=) at 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:743
  #10 0x5660bdb1 in e1000e_start_xmit (core=core@entry=0x7fffeeb754e0, 
txr=, txr@entry=0x7fffbe60) at 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:934
  #11 0x56607e2e in e1000e_set_tctl (core=0x7fffeeb754e0, 
index=, val=) at 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:2431
  #12 0x565f90fd in e1000e_core_write (core=, 
addr=, val=, size=) at 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:3261
  #13 0x55ff4337 in memory_region_write_accessor (mr=, 
addr=, value=, size=, 
shift=, mask=, attrs=...) at 
/home/alxndr/Development/qemu/memory.c:483
  #14 0x55ff3ce0 in access_with_adjusted_size (addr=, 
value=, size=, access_size_min=, 
access_size_max=, access_fn=, mr=0x7fffeeb75110, 
attrs=...) at /home/alxndr/Development/qemu/memory.c:544
  #15 0x55ff3ce0 in memory_region_dispatch_write (mr=, 
addr=, data=0x2b, op=, attrs=...) at 
/home/alxndr/Development/qemu/memory.c:1476

  I can reproduce it in qemu 5.0 built with using:
  cat << EOF | ~/Development/qemu/build/i386-softmmu/qemu-system-i386 -M 
pc-q35-5.0 -netdev user,id=qtest-bn0 -device e1000e,netdev=qtest-bn0 -display 
none -nodefaults -nographic -qtest stdio -monitor none -serial none
  outl 0xcf8 0x8810
  outl 0xcfc 0xe000
  outl 0xcf8 0x8814
  outl 0xcf8 0x8804
  outw 0xcfc 0x7
  outl 0xcf8 0x88a2
  write 0xe420 0x1fc 
0x3ff9ffdf002467ff272d2f3ff9ffdf00246fff272d2f3ff9ffdf002477ff272d2f3ff9ffdf00247fff272d2f3ff9ffdf002487ff272d2f3ff9ffdf00248fff272d2f3ff9ffdf002497ff272d2f3ff9ffdf00249fff272d2f3ff9ffdf0024a7ff272d2f3ff9ffdf0024afff272d2f3ff9ffdf0024b7ff272d2f3ff9ffdf0024bfff272d2f3ff9ffdf0024c7ff272d2f3ff9ffdf0024cfff272d2f3ff9ffdf0024d7ff272d2f3ff9ffdf0024dfff272d2f3ff9ffdf0024e7ff272d2f3ff9ffdf0024efff272d2f3ff9ffdf0024f7ff272d2f3ff9ffdf0024272d2f3ff9ffdf002407ff272d2f3ff9ffdf00240fff272d2f3ff9ffdf002417ff272d2f3ff9ffdf00241fff272d2f3ff9ffdf002427ff272d2f3ff9ffdf00242fff272d2f3ff9ffdf002437ff272d2f3ff9ffdf00243fff272d2f3ff9ffdf002447ff272d2f3ff9ffdf00244fff272d2f3ff9ffdf002457ff272d2f3ff9ffdf00245fff272d2f3ff9ffdf002467ff272d2f3ff9ffdf00246fff27
  write 0xe0b8 0x349 
0xa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa300f52bff003100ffa3

[Bug 1878034] Re: memcpy param-overlap through e1000e_write_to_rx_buffers

[Thomas Huth]

Ok, confirmed, with that new reproducer it also detects the error here
when I compile QEMU with Clang and ASAN enabled.

** Changed in: qemu
   Status: Incomplete => Confirmed

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1878034

Title:
  memcpy param-overlap through e1000e_write_to_rx_buffers

Status in QEMU:
  Confirmed

Bug description:
  Hello,
  While fuzzing, I found an input that triggers an overlapping memcpy (caught 
by AddressSanitizer).
  Overlapping memcpys are undefined behavior according to the POSIX and C 
standards, and can lead to bugs.

  ==22287==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges
  #0 0x563c9f4823d4 in __asan_memcpy 
(/home/alxndr/Development/qemu/build/i386-softmmu/qemu-system-i386+0x97a3d4)
  #1 0x563c9f4cb2b1 in flatview_write_continue 
/home/alxndr/Development/qemu/exec.c:3142:13
  #2 0x563c9f4c3b97 in flatview_write 
/home/alxndr/Development/qemu/exec.c:3177:14
  #3 0x563c9f4c3b97 in address_space_write 
/home/alxndr/Development/qemu/exec.c:3268:18
  #4 0x563c9fbc457b in dma_memory_rw_relaxed 
/home/alxndr/Development/qemu/include/sysemu/dma.h:87:18
  #5 0x563c9fbc457b in dma_memory_rw 
/home/alxndr/Development/qemu/include/sysemu/dma.h:110:12
  #6 0x563c9fbc457b in pci_dma_rw 
/home/alxndr/Development/qemu/include/hw/pci/pci.h:787:5
  #7 0x563c9fbc457b in pci_dma_write 
/home/alxndr/Development/qemu/include/hw/pci/pci.h:800:12
  #8 0x563c9fbc457b in e1000e_write_to_rx_buffers 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:1412:9
  #9 0x563c9fbb9c98 in e1000e_write_packet_to_guest 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:1582:21
  #10 0x563c9fbb9c98 in e1000e_receive_iov 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:1709:9
  #11 0x563c9fba8080 in net_tx_pkt_sendv 
/home/alxndr/Development/qemu/hw/net/net_tx_pkt.c:544:9
  #12 0x563c9fba8080 in net_tx_pkt_send 
/home/alxndr/Development/qemu/hw/net/net_tx_pkt.c:620:9
  #13 0x563c9fba8827 in net_tx_pkt_send_loopback 
/home/alxndr/Development/qemu/hw/net/net_tx_pkt.c:633:11
  #14 0x563c9fbd2052 in e1000e_tx_pkt_send 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:664:16
  #15 0x563c9fbd2052 in e1000e_process_tx_desc 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:743:17
  #16 0x563c9fbd2052 in e1000e_start_xmit 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:934:9
  #17 0x563c9fbcecf0 in e1000e_set_tdt 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:2451:9
  #18 0x563c9fbbf20c in e1000e_core_write 
/home/alxndr/Development/qemu/hw/net/e1000e_core.c:3261:9
  #19 0x563c9f5b68d6 in memory_region_write_accessor 
/home/alxndr/Development/qemu/memory.c:483:5
  #20 0x563c9f5b627f in access_with_adjusted_size 
/home/alxndr/Development/qemu/memory.c:544:18
  #21 0x563c9f5b627f in memory_region_dispatch_write 
/home/alxndr/Development/qemu/memory.c:1476:16

  I can reproduce it in qemu 5.0 built with --enable-sanitizers using:
  cat << EOF | ~/Development/qemu/build/i386-softmmu/qemu-system-i386 -M 
pc-q35-5.0 -accel qtest -qtest stdio -nographic -monitor none -serial none
  outl 0xcf8 0x80001010
  outl 0xcfc 0xe102
  outl 0xcf8 0x80001014
  outl 0xcf8 0x80001004
  outw 0xcfc 0x7
  outl 0xcf8 0x800010a2
  write 0xe102003a 0x3ff 
0xd1055e2d3b0002e101ffd3055e2d3b0002e101ffd5055e2d3b0002e101ffd7055e2d3b0002e101ffd9055e2d3b0002e101ffdb055e2d3b0002e101ffdd055e2d3b0002e101ffdf055e2d3b0002e101ffe1055e2d3b0002e101ffe3055e2d3b0002e101ffe5055e2d3b0002e101ffe7055e2d3b0002e101ffe9055e2d3b0002e101ffeb055e2d3b0002e101ffed055e2d3b0002e101ffef055e2d3b0002e101fff1055e2d3b0002e101fff3055e2d3b0002e101fff5055e2d3b0002e101fff7055e2d3b0002e101fff9055e2d3b0002e101fffb055e2d3b0002e101fffd055e2d3b0002e101055e2d3b0002e101ff01055e2d3b0002e101ff03055e2d3b0002e101ff05055e2d3b0002e101ff07055e2d3b0002e101ff09055e2d3b0002e101ff0b055e2d3b0002e101ff0d055e2d3b0002e101ff0f055e2d3b0002e101ff11055e2d3b0002e101ff13055e2d3b0002e101ff15055e2d3b0002e101ff17055e2d3b0002e101ff19055e2d3b0002e101ff1b055e2d3b0002e101ff1d055e2d3b0002e101ff1f055e2d3b0002e101ff21055e2d3b0002e101ff23055e2d3b0002e101ff25055e2d3b0002e101ff27055e2d3b0002e101ff29055e2d3b0002e101ff2b055e2d3b0002e101ff2d055e2d3b0002e101ff2f055e2d3b0002e101ff31055e2d3b0002e101ff33055e2d3b0002e101ff35055e2d3b0002e101ff37055e2d3b0002e101ff39055e2d3b0002e101ff3b055e2d3b0002e101ff3d055e2d3b0002e101ff3f055e2d3b0002e101ff41055e2d3b0002e101ff43055e2d3b0002e101ff45055e2d3b0002e101ff47055e2d3b0002e101f

Re: [PATCH 05/38] target/riscv: 8-bit Addition & Subtraction Instruction

[LIU Zhiwei]

On 5/24/21 9:00 AM, Palmer Dabbelt wrote:

On Mon, 15 Mar 2021 14:22:58 PDT (-0700), alistai...@gmail.com wrote:
On Fri, Feb 12, 2021 at 10:14 AM LIU Zhiwei  
wrote:


Signed-off-by: LIU Zhiwei 


Acked-by: Alistair Francis 


I saw some reviews on the other ones, but since others (like this) 
just have acks and haven't had any other traffic I'm going to start here.


It looks like the latest spec is 0.9.4, but the changelog is pretty 
minimal between 0.9.5 and 0.9.2:


[0.9.2 -> 0.9.3]

* Changed Zp64 name to Zpsfoperand.
* Added Zprvsfextra for RV64 only instructions.
* Removed SWAP16 encoding. It is an alias of PKBT16.
* Fixed few typos and enhanced precision descriptions on imtermediate 
results.


[0.9.3 -> 0.9.4]

* Fixed few typos and enhanced precision descriptions on imtermediate 
results.

* Fixed/Changed data types for some intrinsic functions.
* Removed "RV32 Only" for Zpsfoperand.

So I'm just going to stick with reviewing based on the latest spec 
 
and try to keep those differences in mind, assuming we're just 
tracking the latest draft here.



Hi Palmer,

It's a good news.

I plan to rebase the patch set and update to the latest specification.

Probably before next week, we can get a v2 patch set.

Zhiwei


Alistair


---
 target/riscv/helper.h   |  9 +++
 target/riscv/insn32.decode  | 11 
 target/riscv/insn_trans/trans_rvp.c.inc | 79 +
 target/riscv/packed_helper.c    | 73 +++
 4 files changed, 172 insertions(+)

diff --git a/target/riscv/helper.h b/target/riscv/helper.h
index 6d622c732a..a69a6b4e84 100644
--- a/target/riscv/helper.h
+++ b/target/riscv/helper.h
@@ -1175,3 +1175,12 @@ DEF_HELPER_3(rstsa16, tl, env, tl, tl)
 DEF_HELPER_3(urstsa16, tl, env, tl, tl)
 DEF_HELPER_3(kstsa16, tl, env, tl, tl)
 DEF_HELPER_3(ukstsa16, tl, env, tl, tl)
+
+DEF_HELPER_3(radd8, tl, env, tl, tl)
+DEF_HELPER_3(uradd8, tl, env, tl, tl)
+DEF_HELPER_3(kadd8, tl, env, tl, tl)
+DEF_HELPER_3(ukadd8, tl, env, tl, tl)
+DEF_HELPER_3(rsub8, tl, env, tl, tl)
+DEF_HELPER_3(ursub8, tl, env, tl, tl)
+DEF_HELPER_3(ksub8, tl, env, tl, tl)
+DEF_HELPER_3(uksub8, tl, env, tl, tl)
diff --git a/target/riscv/insn32.decode b/target/riscv/insn32.decode
index 8815e90476..358dd1fa10 100644
--- a/target/riscv/insn32.decode
+++ b/target/riscv/insn32.decode
@@ -624,3 +624,14 @@ rstsa16    1011011  . . 010 . 
111 @r

 urstsa16   1101011  . . 010 . 111 @r
 kstsa16    1100011  . . 010 . 111 @r
 ukstsa16   1110011  . . 010 . 111 @r
+
+add8   0100100  . . 000 . 111 @r
+radd8  100  . . 000 . 111 @r
+uradd8 0010100  . . 000 . 111 @r
+kadd8  0001100  . . 000 . 111 @r
+ukadd8 0011100  . . 000 . 111 @r
+sub8   0100101  . . 000 . 111 @r
+rsub8  101  . . 000 . 111 @r
+ursub8 0010101  . . 000 . 111 @r
+ksub8  0001101  . . 000 . 111 @r
+uksub8 0011101  . . 000 . 111 @r
diff --git a/target/riscv/insn_trans/trans_rvp.c.inc 
b/target/riscv/insn_trans/trans_rvp.c.inc

index 0885a4fd45..109f560ec9 100644
--- a/target/riscv/insn_trans/trans_rvp.c.inc
+++ b/target/riscv/insn_trans/trans_rvp.c.inc
@@ -159,3 +159,82 @@ GEN_RVP_R_OOL(rstsa16);
 GEN_RVP_R_OOL(urstsa16);
 GEN_RVP_R_OOL(kstsa16);
 GEN_RVP_R_OOL(ukstsa16);
+
+/* 8-bit Addition & Subtraction Instructions */
+/*
+ *  Copied from tcg-op-gvec.c.
+ *
+ *  Perform a vector addition using normal addition and a mask.  
The mask
+ *  should be the sign bit of each lane.  This 6-operation form is 
more
+ *  efficient than separate additions when there are 4 or more 
lanes in

+ *  the 64-bit operation.
+ */
+
+static void gen_simd_add_mask(TCGv d, TCGv a, TCGv b, TCGv m)
+{
+    TCGv t1 = tcg_temp_new();
+    TCGv t2 = tcg_temp_new();
+    TCGv t3 = tcg_temp_new();
+
+    tcg_gen_andc_tl(t1, a, m);
+    tcg_gen_andc_tl(t2, b, m);
+    tcg_gen_xor_tl(t3, a, b);
+    tcg_gen_add_tl(d, t1, t2);
+    tcg_gen_and_tl(t3, t3, m);
+    tcg_gen_xor_tl(d, d, t3);
+
+    tcg_temp_free(t1);
+    tcg_temp_free(t2);
+    tcg_temp_free(t3);
+}
+
+static void tcg_gen_simd_add8(TCGv d, TCGv a, TCGv b)
+{
+    TCGv m = tcg_const_tl((target_ulong)dup_const(MO_8, 0x80));
+    gen_simd_add_mask(d, a, b, m);
+    tcg_temp_free(m);
+}
+
+GEN_RVP_R_INLINE(add8, add, 0, trans_add);
+
+/*
+ *  Copied from tcg-op-gvec.c.
+ *
+ *  Perform a vector subtraction using normal subtraction and a mask.
+ *  Compare gen_addv_mask above.
+ */
+static void gen_simd_sub_mask(TCGv d, TCGv a, TCGv b, TCGv m)
+{
+    TCGv t1 = tcg_temp_new();
+    TCGv t2 = tcg_temp_new();
+    TCGv t3 = tcg_temp_new();
+
+    tcg_gen_or_tl(t1, a, m);
+    tcg_gen_andc_tl(t2, b, m);
+    tcg_gen_eqv_tl(t3, 

Re: [RFC PATCH v4 0/7] Use ACPI PCI hot-plug for Q35

[Michael S. Tsirkin]

On Thu, May 13, 2021 at 08:26:35AM +0200, Julia Suvorova wrote:
> The patch set consists of two parts:
> patches 1-4: introduce new feature
>  'acpi-pci-hotplug-with-bridge-support' on Q35
> patches 5-7: make the feature default along with changes in ACPI tables
> 
> This way maintainers can decide which way to choose without breaking
> the patch set.

This is in good shape, I think the next version can be a non-RFC series.

> With the feature disabled Q35 falls back to the native hot-plug.
> 
> Pros
> * no racy behavior during boot (see 110c477c2ed)
> * eject is possible - according to PCIe spec, attention button
>   press should lead to power off, and then the adapter should be
>   removed manually. As there is no power down state exists in QEMU,
>   we cannot distinguish between an eject and a power down
>   request.
> * no delay during deleting - after the actual power off software
>   must wait at least 1 second before indicating about it. This case
>   is quite important for users, it even has its own bug:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1594168
> * no timer-based behavior - in addition to the previous example,
>   the attention button has a 5-second waiting period, during which
>   the operation can be canceled with a second press. While this
>   looks fine for manual button control, automation will result in
>   the need to queue or drop events, and the software receiving
>   events in all sort of unspecified combinations of attention/power
>   indicator states, which is racy and uppredictable.
> * fixes or reduces the likelihood of the bugs:
> * https://bugzilla.redhat.com/show_bug.cgi?id=1833187
> * https://bugzilla.redhat.com/show_bug.cgi?id=1657077
> * https://bugzilla.redhat.com/show_bug.cgi?id=1669931
> * https://bugzilla.redhat.com/show_bug.cgi?id=1678290
> 
> Cons:
> * no access to possible features presented in slot capabilities
>   (this is only surprise removal AFAIK)
> 
> v4:
> * regain per-port control over hot-plug
> * rebased over acpi-index changes
> * set property on machine type to
>   make pci code more generic [Igor, Michael]
> 
> v3:
> * drop change of _OSC to allow SHPC on hotplugged bridges
> * use 'acpi-root-pci-hotplug'
> * add migration states [Igor]
> * minor style changes
> 
> v2:
> * new ioport range for acpiphp [Gerd]
> * drop find_pci_host() [Igor]
> * explain magic numbers in _OSC [Igor]
> * drop build_q35_pci_hotplug() wrapper [Igor]
> 
> Julia Suvorova (7):
>   hw/acpi/pcihp: Enhance acpi_pcihp_disable_root_bus() to support Q35
>   hw/i386/acpi-build: Add ACPI PCI hot-plug methods to Q35
>   hw/acpi/ich9: Enable ACPI PCI hot-plug
>   hw/pci/pcie: Do not set HPC flag if acpihp is used
>   bios-tables-test: Allow changes in DSDT ACPI tables
>   hw/acpi/ich9: Set ACPI PCI hot-plug as default on Q35
>   bios-tables-test: Update golden binaries
> 
>  hw/i386/acpi-build.h  |   5 +++
>  include/hw/acpi/ich9.h|   5 +++
>  include/hw/acpi/pcihp.h   |   3 +-
>  include/hw/boards.h   |   1 +
>  hw/acpi/ich9.c|  68 ++
>  hw/acpi/pcihp.c   |  22 +++---
>  hw/acpi/piix4.c   |   4 +-
>  hw/core/machine.c |  19 +
>  hw/i386/acpi-build.c  |  32 --
>  hw/i386/pc.c  |   4 +-
>  hw/i386/pc_q35.c  |   8 
>  hw/pci/pcie.c |  11 -
>  tests/data/acpi/q35/DSDT  | Bin 7859 -> 8289 bytes
>  tests/data/acpi/q35/DSDT.acpihmat | Bin 9184 -> 9614 bytes
>  tests/data/acpi/q35/DSDT.bridge   | Bin 7877 -> 11003 bytes
>  tests/data/acpi/q35/DSDT.cphp | Bin 8323 -> 8753 bytes
>  tests/data/acpi/q35/DSDT.dimmpxm  | Bin 9513 -> 9943 bytes
>  tests/data/acpi/q35/DSDT.ipmibt   | Bin 7934 -> 8364 bytes
>  tests/data/acpi/q35/DSDT.memhp| Bin 9218 -> 9648 bytes
>  tests/data/acpi/q35/DSDT.mmio64   | Bin 8990 -> 9419 bytes
>  tests/data/acpi/q35/DSDT.nohpet   | Bin 7717 -> 8147 bytes
>  tests/data/acpi/q35/DSDT.numamem  | Bin 7865 -> 8295 bytes
>  tests/data/acpi/q35/DSDT.tis  | Bin 8465 -> 8894 bytes
>  23 files changed, 161 insertions(+), 21 deletions(-)
> 
> -- 
> 2.30.2

Re: [PATCH 08/38] target/riscv: SIMD 16-bit Compare Instructions

[Palmer Dabbelt]

On Tue, 25 May 2021 22:30:14 PDT (-0700), Palmer Dabbelt wrote:

On Fri, 12 Feb 2021 07:02:26 PST (-0800), zhiwei_...@c-sky.com wrote:

Signed-off-by: LIU Zhiwei 
---
 target/riscv/helper.h   |  6 
 target/riscv/insn32.decode  |  6 
 target/riscv/insn_trans/trans_rvp.c.inc |  7 
 target/riscv/packed_helper.c| 46 +
 4 files changed, 65 insertions(+)

diff --git a/target/riscv/helper.h b/target/riscv/helper.h
index 0ecd4d53f9..f41f9a 100644
--- a/target/riscv/helper.h
+++ b/target/riscv/helper.h
@@ -1202,3 +1202,9 @@ DEF_HELPER_3(sll8, tl, env, tl, tl)
 DEF_HELPER_3(ksll8, tl, env, tl, tl)
 DEF_HELPER_3(kslra8, tl, env, tl, tl)
 DEF_HELPER_3(kslra8_u, tl, env, tl, tl)
+
+DEF_HELPER_3(cmpeq16, tl, env, tl, tl)
+DEF_HELPER_3(scmplt16, tl, env, tl, tl)
+DEF_HELPER_3(scmple16, tl, env, tl, tl)
+DEF_HELPER_3(ucmplt16, tl, env, tl, tl)
+DEF_HELPER_3(ucmple16, tl, env, tl, tl)
diff --git a/target/riscv/insn32.decode b/target/riscv/insn32.decode
index cc782fcde5..f3cd508396 100644
--- a/target/riscv/insn32.decode
+++ b/target/riscv/insn32.decode
@@ -669,3 +669,9 @@ ksll8  0110110  . . 000 . 111 @r
 kslli8 010  01... . 000 . 111 @sh3
 kslra8 010  . . 000 . 111 @r
 kslra8_u   0110111  . . 000 . 111 @r
+
+cmpeq160100110  . . 000 . 111 @r
+scmplt16   110  . . 000 . 111 @r
+scmple16   0001110  . . 000 . 111 @r
+ucmplt16   0010110  . . 000 . 111 @r
+ucmple16   000  . . 000 . 111 @r
diff --git a/target/riscv/insn_trans/trans_rvp.c.inc 
b/target/riscv/insn_trans/trans_rvp.c.inc
index 12a64849eb..6438dfb776 100644
--- a/target/riscv/insn_trans/trans_rvp.c.inc
+++ b/target/riscv/insn_trans/trans_rvp.c.inc
@@ -369,3 +369,10 @@ GEN_RVP_SHIFTI(slli8, sll8, tcg_gen_vec_shl8i_i64);
 GEN_RVP_SHIFTI(srai8_u, sra8_u, NULL);
 GEN_RVP_SHIFTI(srli8_u, srl8_u, NULL);
 GEN_RVP_SHIFTI(kslli8, ksll8, NULL);
+
+/* SIMD 16-bit Compare Instructions */
+GEN_RVP_R_OOL(cmpeq16);
+GEN_RVP_R_OOL(scmplt16);
+GEN_RVP_R_OOL(scmple16);
+GEN_RVP_R_OOL(ucmplt16);
+GEN_RVP_R_OOL(ucmple16);
diff --git a/target/riscv/packed_helper.c b/target/riscv/packed_helper.c
index ab9ebc472b..30b916b5ad 100644
--- a/target/riscv/packed_helper.c
+++ b/target/riscv/packed_helper.c
@@ -631,3 +631,49 @@ static inline void do_kslra8_u(CPURISCVState *env, void 
*vd, void *va,
 }

 RVPR(kslra8_u, 1, 1);
+
+/* SIMD 16-bit Compare Instructions */
+static inline void do_cmpeq16(CPURISCVState *env, void *vd, void *va,
+  void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] == b[i]) ? 0x : 0x0;
+}
+
+RVPR(cmpeq16, 1, 2);
+
+static inline void do_scmplt16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+int16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] < b[i]) ? 0x : 0x0;
+}
+
+RVPR(scmplt16, 1, 2);
+
+static inline void do_scmple16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+int16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] <= b[i]) ? 0x : 0x0;
+}
+
+RVPR(scmple16, 1, 2);
+
+static inline void do_ucmplt16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] < b[i]) ? 0x : 0x0;
+}
+
+RVPR(ucmplt16, 1, 2);
+
+static inline void do_ucmple16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] <= b[i]) ? 0x : 0x0;
+}
+
+RVPR(ucmple16, 1, 2);


Thanks, this is on for-next.


Oops, got my threads crossed.

Re: [PATCH 08/38] target/riscv: SIMD 16-bit Compare Instructions

[Palmer Dabbelt]

On Fri, 12 Feb 2021 07:02:26 PST (-0800), zhiwei_...@c-sky.com wrote:

Signed-off-by: LIU Zhiwei 
---
 target/riscv/helper.h   |  6 
 target/riscv/insn32.decode  |  6 
 target/riscv/insn_trans/trans_rvp.c.inc |  7 
 target/riscv/packed_helper.c| 46 +
 4 files changed, 65 insertions(+)

diff --git a/target/riscv/helper.h b/target/riscv/helper.h
index 0ecd4d53f9..f41f9a 100644
--- a/target/riscv/helper.h
+++ b/target/riscv/helper.h
@@ -1202,3 +1202,9 @@ DEF_HELPER_3(sll8, tl, env, tl, tl)
 DEF_HELPER_3(ksll8, tl, env, tl, tl)
 DEF_HELPER_3(kslra8, tl, env, tl, tl)
 DEF_HELPER_3(kslra8_u, tl, env, tl, tl)
+
+DEF_HELPER_3(cmpeq16, tl, env, tl, tl)
+DEF_HELPER_3(scmplt16, tl, env, tl, tl)
+DEF_HELPER_3(scmple16, tl, env, tl, tl)
+DEF_HELPER_3(ucmplt16, tl, env, tl, tl)
+DEF_HELPER_3(ucmple16, tl, env, tl, tl)
diff --git a/target/riscv/insn32.decode b/target/riscv/insn32.decode
index cc782fcde5..f3cd508396 100644
--- a/target/riscv/insn32.decode
+++ b/target/riscv/insn32.decode
@@ -669,3 +669,9 @@ ksll8  0110110  . . 000 . 111 @r
 kslli8 010  01... . 000 . 111 @sh3
 kslra8 010  . . 000 . 111 @r
 kslra8_u   0110111  . . 000 . 111 @r
+
+cmpeq160100110  . . 000 . 111 @r
+scmplt16   110  . . 000 . 111 @r
+scmple16   0001110  . . 000 . 111 @r
+ucmplt16   0010110  . . 000 . 111 @r
+ucmple16   000  . . 000 . 111 @r
diff --git a/target/riscv/insn_trans/trans_rvp.c.inc 
b/target/riscv/insn_trans/trans_rvp.c.inc
index 12a64849eb..6438dfb776 100644
--- a/target/riscv/insn_trans/trans_rvp.c.inc
+++ b/target/riscv/insn_trans/trans_rvp.c.inc
@@ -369,3 +369,10 @@ GEN_RVP_SHIFTI(slli8, sll8, tcg_gen_vec_shl8i_i64);
 GEN_RVP_SHIFTI(srai8_u, sra8_u, NULL);
 GEN_RVP_SHIFTI(srli8_u, srl8_u, NULL);
 GEN_RVP_SHIFTI(kslli8, ksll8, NULL);
+
+/* SIMD 16-bit Compare Instructions */
+GEN_RVP_R_OOL(cmpeq16);
+GEN_RVP_R_OOL(scmplt16);
+GEN_RVP_R_OOL(scmple16);
+GEN_RVP_R_OOL(ucmplt16);
+GEN_RVP_R_OOL(ucmple16);
diff --git a/target/riscv/packed_helper.c b/target/riscv/packed_helper.c
index ab9ebc472b..30b916b5ad 100644
--- a/target/riscv/packed_helper.c
+++ b/target/riscv/packed_helper.c
@@ -631,3 +631,49 @@ static inline void do_kslra8_u(CPURISCVState *env, void 
*vd, void *va,
 }

 RVPR(kslra8_u, 1, 1);
+
+/* SIMD 16-bit Compare Instructions */
+static inline void do_cmpeq16(CPURISCVState *env, void *vd, void *va,
+  void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] == b[i]) ? 0x : 0x0;
+}
+
+RVPR(cmpeq16, 1, 2);
+
+static inline void do_scmplt16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+int16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] < b[i]) ? 0x : 0x0;
+}
+
+RVPR(scmplt16, 1, 2);
+
+static inline void do_scmple16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+int16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] <= b[i]) ? 0x : 0x0;
+}
+
+RVPR(scmple16, 1, 2);
+
+static inline void do_ucmplt16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] < b[i]) ? 0x : 0x0;
+}
+
+RVPR(ucmplt16, 1, 2);
+
+static inline void do_ucmple16(CPURISCVState *env, void *vd, void *va,
+   void *vb, uint8_t i)
+{
+uint16_t *d = vd, *a = va, *b = vb;
+d[i] = (a[i] <= b[i]) ? 0x : 0x0;
+}
+
+RVPR(ucmple16, 1, 2);


Thanks, this is on for-next.

[PATCH] linux-user: Remove unnecessary static assert involving __SIGRTMAX

[Michael Forney]

Since "linux-user: fix use of SIGRTMIN" (6bc024e7), qemu removed
use of __SIGRTMAX except for in this QEMU_BUILD_BUG_ON assert.
Presumably, this check is to ensure that the loop in signal_table_init
from SIGRTMIN to SIGRTMAX falls within the bounds of
host_to_target_signal_table (_NSIG).

However, _NSIG is already defined to be the one larger than the
largest signal supported by the system (as specified in the upcoming
POSIX revision[0]), so the check is unnecessary.

musl libc does not define __SIGRTMAX, so removing this check fixes
one of the last remaining errors when building qemu.

[0] https://www.austingroupbugs.net/view.php?id=741

Signed-off-by: Michael Forney 
---
If you prefer, I can send an alternate patch to leave the
QEMU_BUILD_BUG_ON, but guard it by #ifdef __SIGRTMAX.

 linux-user/signal.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/linux-user/signal.c b/linux-user/signal.c
index 9016896dcd..6f62f2b528 100644
--- a/linux-user/signal.c
+++ b/linux-user/signal.c
@@ -38,7 +38,6 @@ static void host_signal_handler(int host_signum, siginfo_t 
*info,
  * Signal number 0 is reserved for use as kill(pid, 0), to test whether
  * a process exists without sending it a signal.
  */
-QEMU_BUILD_BUG_ON(__SIGRTMAX + 1 != _NSIG);
 static uint8_t host_to_target_signal_table[_NSIG] = {
 [SIGHUP] = TARGET_SIGHUP,
 [SIGINT] = TARGET_SIGINT,
-- 
2.31.1

[PATCH] linux-user: Use public sigev_notify_thread_id member if available

[Michael Forney]

_sigev_un._tid is an internal glibc field and is not available on
musl libc. The sigevent(7) man page and Linux UAPI headers both use
sigev_notify_thread_id as a public way to access this field.

musl libc supports this field since 1.2.2[0], and glibc plans to
add support as well[1][2].

If sigev_notify_thread_id is not available, fall back to _sigev_un._tid
as before.

[0] 
http://git.musl-libc.org/cgit/musl/commit/?id=7c71792e87691451f2a6b76348e83ad1889f1dcb
[1] https://www.openwall.com/lists/musl/2019/08/01/5
[2] https://sourceware.org/bugzilla/show_bug.cgi?id=27417

Signed-off-by: Michael Forney 
---
 configure| 16 
 linux-user/syscall.c |  6 +-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 676239c697..fa39b0a727 100755
--- a/configure
+++ b/configure
@@ -4462,6 +4462,19 @@ if compile_prog "" "" ; then
 st_atim=yes
 fi
 
+##
+# check if we have sigev_notify_thread_id
+
+sigev_notify_thread_id=no
+cat > $TMPC << EOF
+#include 
+#include 
+int main(void) { return offsetof(struct sigevent, sigev_notify_thread_id); }
+EOF
+if compile_prog "" "" ; then
+sigev_notify_thread_id=yes
+fi
+
 ##
 # check if trace backend exists
 
@@ -5718,6 +5731,9 @@ fi
 if test "$st_atim" = "yes" ; then
   echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak
 fi
+if test "$sigev_notify_thread_id" = "yes" ; then
+  echo "HAVE_SIGEV_NOTIFY_THREAD_ID=y" >> $config_host_mak
+fi
 if test "$byteswap_h" = "yes" ; then
   echo "CONFIG_BYTESWAP_H=y" >> $config_host_mak
 fi
diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index c9f812091c..63464f9a96 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -7393,6 +7393,10 @@ static inline abi_long host_to_target_timex64(abi_long 
target_addr,
 }
 #endif
 
+#ifndef HAVE_SIGEV_NOTIFY_THREAD_ID
+#define sigev_notify_thread_id _sigev_un._tid
+#endif
+
 static inline abi_long target_to_host_sigevent(struct sigevent *host_sevp,
abi_ulong target_addr)
 {
@@ -7413,7 +7417,7 @@ static inline abi_long target_to_host_sigevent(struct 
sigevent *host_sevp,
 host_sevp->sigev_signo =
 target_to_host_signal(tswap32(target_sevp->sigev_signo));
 host_sevp->sigev_notify = tswap32(target_sevp->sigev_notify);
-host_sevp->_sigev_un._tid = tswap32(target_sevp->_sigev_un._tid);
+host_sevp->sigev_notify_thread_id = tswap32(target_sevp->_sigev_un._tid);
 
 unlock_user_struct(target_sevp, target_addr, 1);
 return 0;
-- 
2.31.1

[PATCH] linux-user: Fix incorrect use of feature-test-macros

[Michael Forney]

The _POSIX_C_SOURCE and _XOPEN_SOURCE macros are used by the
application to indicate to libc which declarations it should expose.
Since qemu does not define them anywhere, it does not make sense
to check their value.

Instead, since the intent is to determine whether the host struct
stat supports the st_*tim fields, use the configure test result
which does exactly that.

Signed-off-by: Michael Forney 
---
 linux-user/syscall.c | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index c9f812091c..9a52f235d2 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -7470,7 +7470,7 @@ static inline abi_long host_to_target_stat64(void 
*cpu_env,
 __put_user(host_st->st_atime, &target_st->target_st_atime);
 __put_user(host_st->st_mtime, &target_st->target_st_mtime);
 __put_user(host_st->st_ctime, &target_st->target_st_ctime);
-#if _POSIX_C_SOURCE >= 200809L || _XOPEN_SOURCE >= 700
+#ifdef HAVE_STRUCT_STAT_ST_ATIM
 __put_user(host_st->st_atim.tv_nsec, &target_st->target_st_atime_nsec);
 __put_user(host_st->st_mtim.tv_nsec, &target_st->target_st_mtime_nsec);
 __put_user(host_st->st_ctim.tv_nsec, &target_st->target_st_ctime_nsec);
@@ -7505,7 +7505,7 @@ static inline abi_long host_to_target_stat64(void 
*cpu_env,
 __put_user(host_st->st_atime, &target_st->target_st_atime);
 __put_user(host_st->st_mtime, &target_st->target_st_mtime);
 __put_user(host_st->st_ctime, &target_st->target_st_ctime);
-#if _POSIX_C_SOURCE >= 200809L || _XOPEN_SOURCE >= 700
+#ifdef HAVE_STRUCT_STAT_ST_ATIM
 __put_user(host_st->st_atim.tv_nsec, &target_st->target_st_atime_nsec);
 __put_user(host_st->st_mtim.tv_nsec, &target_st->target_st_mtime_nsec);
 __put_user(host_st->st_ctim.tv_nsec, &target_st->target_st_ctime_nsec);
@@ -10056,8 +10056,7 @@ static abi_long do_syscall1(void *cpu_env, int num, 
abi_long arg1,
 __put_user(st.st_atime, &target_st->target_st_atime);
 __put_user(st.st_mtime, &target_st->target_st_mtime);
 __put_user(st.st_ctime, &target_st->target_st_ctime);
-#if (_POSIX_C_SOURCE >= 200809L || _XOPEN_SOURCE >= 700) && \
-defined(TARGET_STAT_HAVE_NSEC)
+#if defined(HAVE_STRUCT_STAT_ST_ATIM) && defined(TARGET_STAT_HAVE_NSEC)
 __put_user(st.st_atim.tv_nsec,
&target_st->target_st_atime_nsec);
 __put_user(st.st_mtim.tv_nsec,
-- 
2.31.1

[RFC PATCH v4] ppc/spapr: Add support for H_SCM_PERFORMANCE_STATS hcall

[Vaibhav Jain]

Add support for H_SCM_PERFORMANCE_STATS described at [1] for
spapr nvdimms. This enables guest to fetch performance stats[2] like
expected life of an nvdimm ('MemLife ') etc and display them to the
user. Linux kernel support for fetching these performance stats and
exposing them to the user-space was done via [3].

The hcall semantics mandate that each nvdimm performance stats is
uniquely identied by a 8-byte ascii string encoded as an unsigned
integer (e.g 'MemLife ' == 0x4D656D4C69666520) and its value be a
8-byte unsigned integer. These performance-stats are exchanged with
the guest in via a guest allocated buffer called
'requestAndResultBuffer' or rr-buffer for short. This buffer contains
a header descibed by 'struct papr_scm_perf_stats' followed by an array
of performance-stats described by 'struct papr_scm_perf_stat'. The
hypervisor is expected to validate the rr-buffer header and then based
on the request copy the needed performance-stats to the array of
'struct papr_scm_perf_stat' following the header.

The patch proposes a new function h_scm_performance_stats() that
services the H_SCM_PERFORMANCE_STATS hcall. After verifying the
validity of the rr-buffer header via scm_perf_check_rr_buffer() it
proceeds to fill the rr-buffer with requested performance-stats. The
value of individual stats is retrived from individual accessor
function for the stat which are indexed in the array
'nvdimm_perf_stats'.

References:
[1] "Hypercall Op-codes (hcalls)"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/powerpc/papr_hcalls.rst#n269
[2] Sysfs attribute documentation for papr_scm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/ABI/testing/sysfs-bus-papr-pmem#n36
[3] "powerpc/papr_scm: Fetch nvdimm performance stats from PHYP"
https://lore.kernel.org/r/20200731064153.182203-2-vaib...@linux.ibm.com

Signed-off-by: Vaibhav Jain 
---
Changelog

Since RFC-v3:
* Introduced consistent types 'perf_stat_id/val' for handling
  perf-stats in code rather than switching between u64 and char[8]. [
  David Gibson ]
* Removed a redundant temporary variable 'stat_id' usage from
  nvdimm_stat_getval() [ David Gibson ]
* Move the tests for buffer sizes around to a a central place in
  h_scm_performance_stats() [ David Gibson ]
* Update the allocation size/type of rr-buffer in
  h_scm_performance_stats [ David Gibson, Groug ]
* Reworked the loop for generating canned-response in
  h_scm_performance_stats().
* Reworked logic for returning an unknown stat-id in h_scm_performance_stats().

Since RFC-v2:
* s/SCM_STATS_MAX_OUTPUT_BUFFER/SCM_STATS_MIN_OUTPUT_BUFFER/ thats the
  minimum size buffer needed to return all supported performance
  stats. Value for this is derived from size of array 'nvdimm_perf_stats'
* Added SCM_STATS_MAX_OUTPUT_BUFFER to indicate maximum supported
  rr-buffer size from a guest. Value for this is derived from hard
  limit of SCM_STATS_MAX_STATS.
* Updated scm_perf_check_rr_buffer() to add a check for max size of
  rr-buffer. [David]
* Updated scm_perf_check_rr_buffer() to removed a reduntant check for
  min size of rr-buffer [David]
* Updated h_scm_performance_stats() to have a single allocation for
  rr-buffer and copy it back to guest memory in a single shot. [David]

Since RFC-v1:
* Removed empty lines from code [ David ]
* Updated struct papr_scm_perf_stat to use uint64_t for
  statistic_id.
* Added a hard limit on max number of stats requested to 255 [ David ]
* Updated scm_perf_check_rr_buffer() to check for rr-buffer header
  size [ David ]
* Removed a redundant check from nvdimm_stat_getval() [ David ]
* Removed a redundant call to address_space_access_valid() in
  scm_perf_check_rr_buffer() [ David ]
* Instead of allocating a minimum size local buffer, allocate a max
  possible size local rr-buffer. [ David ]
* Updated nvdimm_stat_getval() to set 'val' to '0' on error. [ David ]
* Updated h_scm_performance_stats() to use a canned-response method
  for simplifying num_stats==0 case [ David ].
---
 hw/ppc/spapr_nvdimm.c  | 219 +
 include/hw/ppc/spapr.h |  22 -
 2 files changed, 240 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_nvdimm.c b/hw/ppc/spapr_nvdimm.c
index 252204e25f..81ab2811bc 100644
--- a/hw/ppc/spapr_nvdimm.c
+++ b/hw/ppc/spapr_nvdimm.c
@@ -35,6 +35,23 @@
 /* SCM device is unable to persist memory contents */
 #define PAPR_PMEM_UNARMED PPC_BIT(0)
 
+/* Maximum number of stats that we can return back in a single stat request */
+#define SCM_STATS_MAX_STATS 255
+
+/* Given _numstats_ != 0 calculate the size of RR buffer required */
+#define SCM_STATS_RR_BUFFER_SIZE(_numstats_)\
+(sizeof(struct papr_scm_perf_stats) +   \
+ sizeof(struct papr_scm_perf_stat) *\
+ (_numstats_))
+
+/* Maximum possible output buffer to fulfill a perf-stats request */
+#define SCM_STATS_MAX

[PATCH V7 4/6] net/colo-compare: Move data structure and define to .h file.

[Zhang Chen]

Rename structure with COLO index and move it to .h file,
It make other modules can reuse COLO code.

Signed-off-by: Zhang Chen 
---
 net/colo-compare.c | 132 -
 net/colo-compare.h |  86 +
 2 files changed, 109 insertions(+), 109 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 9d1ad99941..8175612c5f 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -17,29 +17,18 @@
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "qapi/error.h"
-#include "net/net.h"
 #include "net/eth.h"
 #include "qom/object_interfaces.h"
 #include "qemu/iov.h"
 #include "qom/object.h"
 #include "net/queue.h"
-#include "chardev/char-fe.h"
 #include "qemu/sockets.h"
-#include "colo.h"
-#include "sysemu/iothread.h"
 #include "net/colo-compare.h"
-#include "migration/colo.h"
-#include "migration/migration.h"
 #include "util.h"
 
 #include "block/aio-wait.h"
 #include "qemu/coroutine.h"
 
-#define TYPE_COLO_COMPARE "colo-compare"
-typedef struct CompareState CompareState;
-DECLARE_INSTANCE_CHECKER(CompareState, COLO_COMPARE,
- TYPE_COLO_COMPARE)
-
 static QTAILQ_HEAD(, CompareState) net_compares =
QTAILQ_HEAD_INITIALIZER(net_compares);
 
@@ -47,13 +36,13 @@ static NotifierList colo_compare_notifiers =
 NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers);
 
 #define COMPARE_READ_LEN_MAX NET_BUFSIZE
-#define MAX_QUEUE_SIZE 1024
+#define MAX_COLO_QUEUE_SIZE 1024
 
 #define COLO_COMPARE_FREE_PRIMARY 0x01
 #define COLO_COMPARE_FREE_SECONDARY   0x02
 
-#define REGULAR_PACKET_CHECK_MS 1000
-#define DEFAULT_TIME_OUT_MS 3000
+#define COLO_REGULAR_PACKET_CHECK_MS 1000
+#define COLO_DEFAULT_TIME_OUT_MS 3000
 
 /* #define DEBUG_COLO_PACKETS */
 
@@ -64,87 +53,6 @@ static QemuCond event_complete_cond;
 static int event_unhandled_count;
 static uint32_t max_queue_size;
 
-/*
- *  + CompareState ++
- *  |   |
- *  +---+   +---+ +---+
- *  |   conn list   + - >  conn + --- >  conn + -- > ..
- *  +---+   +---+ +---+
- *  |   | |   | |  |
- *  +---+ +---v+  +---v++---v+ +---v+
- *|primary |  |secondary|primary | |secondary
- *|packet  |  |packet  +|packet  | |packet  +
- *++  ++++ ++
- *|   | |  |
- *+---v+  +---v++---v+ +---v+
- *|primary |  |secondary|primary | |secondary
- *|packet  |  |packet  +|packet  | |packet  +
- *++  ++++ ++
- *|   | |  |
- *+---v+  +---v++---v+ +---v+
- *|primary |  |secondary|primary | |secondary
- *|packet  |  |packet  +|packet  | |packet  +
- *++  ++++ ++
- */
-
-typedef struct SendCo {
-Coroutine *co;
-struct CompareState *s;
-CharBackend *chr;
-GQueue send_list;
-bool notify_remote_frame;
-bool done;
-int ret;
-} SendCo;
-
-typedef struct SendEntry {
-uint32_t size;
-uint32_t vnet_hdr_len;
-uint8_t *buf;
-} SendEntry;
-
-struct CompareState {
-Object parent;
-
-char *pri_indev;
-char *sec_indev;
-char *outdev;
-char *notify_dev;
-CharBackend chr_pri_in;
-CharBackend chr_sec_in;
-CharBackend chr_out;
-CharBackend chr_notify_dev;
-SocketReadState pri_rs;
-SocketReadState sec_rs;
-SocketReadState notify_rs;
-SendCo out_sendco;
-SendCo notify_sendco;
-bool vnet_hdr;
-uint64_t compare_timeout;
-uint32_t expired_scan_cycle;
-
-/*
- * Record the connection that through the NIC
- * Element type: Connection
- */
-GQueue conn_list;
-/* Record the connection without repetition */
-GHashTable *connection_track_table;
-
-IOThread *iothread;
-GMainContext *worker_context;
-QEMUTimer *packet_check_timer;
-
-QEMUBH *event_bh;
-enum colo_event event;
-
-QTAILQ_ENTRY(CompareState) next;
-};
-
-typedef struct CompareClass {
-ObjectClass parent_class;
-} CompareClass;
-
 enum {
 PRIMARY_IN = 0,
 SECONDARY_IN,
@@ -155,6 +63,12 @@ static const char *colo_mode[] = {
 [SECONDARY_IN] = "secondary",
 };
 
+typedef struct COLOSendEntry {
+uint32_t size;
+uint32_t vnet_hdr_len;
+uint8_t *buf;
+} COLOSendEntry;
+
 static int compare_chr_send(CompareState *s,
 uint8_t *buf,
 uint32_t size,
@@ -737,19 +651,19 @@ static void colo_compare_connection(void *opaque, void 
*user_data)
 
 

[PATCH V7 2/6] util/qemu-sockets.c: Add inet_parse_base to handle InetSocketAddressBase

[Zhang Chen]

No need to carry the flag all the time in many scenarios.

Signed-off-by: Zhang Chen 
---
 include/qemu/sockets.h |  1 +
 util/qemu-sockets.c| 14 ++
 2 files changed, 15 insertions(+)

diff --git a/include/qemu/sockets.h b/include/qemu/sockets.h
index 7d1f813576..d5abc227eb 100644
--- a/include/qemu/sockets.h
+++ b/include/qemu/sockets.h
@@ -32,6 +32,7 @@ int socket_set_fast_reuse(int fd);
 int inet_ai_family_from_address(InetSocketAddress *addr,
 Error **errp);
 int inet_parse(InetSocketAddress *addr, const char *str, Error **errp);
+int inet_parse_base(InetSocketAddressBase *addr, const char *str, Error 
**errp);
 int inet_connect(const char *str, Error **errp);
 int inet_connect_saddr(InetSocketAddress *saddr, Error **errp);
 
diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
index 8af0278f15..c16eba1917 100644
--- a/util/qemu-sockets.c
+++ b/util/qemu-sockets.c
@@ -690,6 +690,20 @@ int inet_parse(InetSocketAddress *addr, const char *str, 
Error **errp)
 return 0;
 }
 
+int inet_parse_base(InetSocketAddressBase *base, const char *str, Error **errp)
+{
+InetSocketAddress *addr;
+int ret = 0;
+
+addr = g_new0(InetSocketAddress, 1);
+ret = inet_parse(addr, str, errp);
+
+base->host = addr->host;
+base->port = addr->port;
+
+g_free(addr);
+return ret;
+}
 
 /**
  * Create a blocking socket and connect it to an address.
-- 
2.25.1

[PATCH V7 6/6] net/net.c: Add handler for COLO passthrough connection

[Zhang Chen]

Use connection protocol,src port,dst port,src ip,dst ip as the key
to bypass certain network traffic in COLO compare.

Signed-off-by: Zhang Chen 
---
 net/net.c | 162 +-
 1 file changed, 160 insertions(+), 2 deletions(-)

diff --git a/net/net.c b/net/net.c
index a4bee86b24..6cb0387199 100644
--- a/net/net.c
+++ b/net/net.c
@@ -54,6 +54,8 @@
 #include "sysemu/runstate.h"
 #include "net/filter.h"
 #include "qapi/string-output-visitor.h"
+#include "net/colo-compare.h"
+#include "qom/object_interfaces.h"
 
 /* Net bridge is currently not supported for W32. */
 #if !defined(_WIN32)
@@ -1194,14 +1196,170 @@ void qmp_netdev_del(const char *id, Error **errp)
 }
 }
 
+static CompareState *colo_passthrough_check(IPFlowSpec *spec, Error **errp)
+{
+Object *container;
+Object *obj;
+CompareState *s;
+
+if (!spec->object_name) {
+error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "object-name",
+   "Need input colo-compare object name");
+return NULL;
+}
+
+container = object_get_objects_root();
+obj = object_resolve_path_component(container, spec->object_name);
+if (!obj) {
+error_setg(errp, "colo-compare '%s' not found", spec->object_name);
+return NULL;
+}
+
+s = COLO_COMPARE(obj);
+
+if (!getprotobyname(spec->protocol)) {
+error_setg(errp, "COLO pass through get wrong protocol");
+return NULL;
+}
+
+if ((spec->source->host && !qemu_isdigit(spec->source->host[0])) ||
+(spec->destination->host &&
+!qemu_isdigit(spec->destination->host[0]))) {
+error_setg(errp, "COLO pass through get wrong IP");
+return NULL;
+}
+
+if (atoi(spec->source->port) > 65536 || atoi(spec->source->port) < 0 ||
+atoi(spec->destination->port) > 65536 ||
+atoi(spec->destination->port) < 0) {
+error_setg(errp, "COLO pass through get wrong port");
+return NULL;
+}
+
+return s;
+}
+
+static COLOPassthroughEntry *compare_passthrough_find(CompareState *s,
+  COLOPassthroughEntry 
*ent)
+{
+COLOPassthroughEntry *next = NULL, *origin = NULL;
+
+if (!QLIST_EMPTY(&s->passthroughlist)) {
+QLIST_FOREACH_SAFE(origin, &s->passthroughlist, node, next) {
+if ((ent->l4_protocol->p_proto == origin->l4_protocol->p_proto) &&
+(ent->src_port == origin->src_port) &&
+(ent->dst_port == origin->dst_port) &&
+(ent->src_ip.s_addr == origin->src_ip.s_addr) &&
+(ent->dst_ip.s_addr == origin->dst_ip.s_addr)) {
+return origin;
+}
+}
+}
+
+return NULL;
+}
+
+static void compare_passthrough_add(CompareState *s,
+IPFlowSpec *spec,
+Error **errp)
+{
+COLOPassthroughEntry *pass = NULL;
+
+pass = g_new0(COLOPassthroughEntry, 1);
+
+pass->l4_protocol = getprotobyname(spec->protocol);
+pass->src_port = atoi(spec->source->port);
+pass->dst_port = atoi(spec->destination->port);
+
+if (!inet_aton(spec->source->host, &pass->src_ip)) {
+pass->src_ip.s_addr = 0;
+}
+
+if (!inet_aton(spec->destination->host, &pass->dst_ip)) {
+pass->dst_ip.s_addr = 0;
+}
+
+qemu_mutex_lock(&s->passthroughlist_mutex);
+if (compare_passthrough_find(s, pass)) {
+error_setg(errp, "The pass through connection already exists");
+g_free(pass);
+qemu_mutex_unlock(&s->passthroughlist_mutex);
+return;
+}
+
+QLIST_INSERT_HEAD(&s->passthroughlist, pass, node);
+qemu_mutex_unlock(&s->passthroughlist_mutex);
+}
+
+static void compare_passthrough_del(CompareState *s,
+IPFlowSpec *spec,
+Error **errp)
+{
+COLOPassthroughEntry *pass = NULL, *result = NULL;
+
+pass = g_new0(COLOPassthroughEntry, 1);
+
+pass->l4_protocol = getprotobyname(spec->protocol);
+pass->src_port = atoi(spec->source->port);
+pass->dst_port = atoi(spec->destination->port);
+
+if (!inet_aton(spec->source->host, &pass->src_ip)) {
+pass->src_ip.s_addr = 0;
+}
+
+if (!inet_aton(spec->destination->host, &pass->dst_ip)) {
+pass->dst_ip.s_addr = 0;
+}
+
+qemu_mutex_lock(&s->passthroughlist_mutex);
+
+result = compare_passthrough_find(s, pass);
+if (result) {
+QLIST_REMOVE(result, node);
+g_free(result);
+} else {
+error_setg(errp, "Can't find the IP flow Spec");
+}
+
+g_free(pass);
+qemu_mutex_unlock(&s->passthroughlist_mutex);
+}
+
+
 void qmp_colo_passthrough_add(IPFlowSpec *spec, Error **errp)
 {
-/* TODO implement setup passthrough rule */
+CompareState *s;
+Error *err = NULL;
+
+s = colo_passthrough_check(spec, &err);
+if (err) {
+error_propagate(errp, 

[PATCH V7 3/6] hmp-commands: Add new HMP command for COLO passthrough

[Zhang Chen]

Add hmp_colo_passthrough_add and hmp_colo_passthrough_del make user
can maintain COLO network passthrough list in human monitor

Signed-off-by: Zhang Chen 
---
 hmp-commands.hx   | 26 ++
 include/monitor/hmp.h |  2 ++
 monitor/hmp-cmds.c| 82 +++
 3 files changed, 110 insertions(+)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index 435c591a1c..cbb08623c7 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1290,6 +1290,32 @@ SRST
   Remove host network device.
 ERST
 
+{
+.name   = "colo_passthrough_add",
+.args_type  = "protocol:s,object-name:s?,src:s?,dst:s?",
+.params = "protocol [object-name] [src] [dst]",
+.help   = "Add network stream to colo passthrough list",
+.cmd= hmp_colo_passthrough_add,
+},
+
+SRST
+``colo_passthrough_add``
+  Add network stream to colo passthrough list.
+ERST
+
+{
+.name   = "colo_passthrough_del",
+.args_type  = "protocol:s,object-name:s?,src:s?,dst:s?",
+.params = "protocol [object-name] [src] [dst]",
+.help   = "Delete network stream from colo passthrough list",
+.cmd= hmp_colo_passthrough_del,
+},
+
+SRST
+``colo_passthrough_del``
+  Delete network stream from colo passthrough list.
+ERST
+
 {
 .name   = "object_add",
 .args_type  = "object:S",
diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
index 605d57287a..a784f98531 100644
--- a/include/monitor/hmp.h
+++ b/include/monitor/hmp.h
@@ -77,6 +77,8 @@ void hmp_device_del(Monitor *mon, const QDict *qdict);
 void hmp_dump_guest_memory(Monitor *mon, const QDict *qdict);
 void hmp_netdev_add(Monitor *mon, const QDict *qdict);
 void hmp_netdev_del(Monitor *mon, const QDict *qdict);
+void hmp_colo_passthrough_add(Monitor *mon, const QDict *qdict);
+void hmp_colo_passthrough_del(Monitor *mon, const QDict *qdict);
 void hmp_getfd(Monitor *mon, const QDict *qdict);
 void hmp_closefd(Monitor *mon, const QDict *qdict);
 void hmp_sendkey(Monitor *mon, const QDict *qdict);
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index d9bef63373..01b467e306 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -1634,6 +1634,88 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 hmp_handle_error(mon, err);
 }
 
+void hmp_colo_passthrough_add(Monitor *mon, const QDict *qdict)
+{
+IPFlowSpec *spec = g_new0(IPFlowSpec, 1);
+char *src, *dst;
+Error *err = NULL;
+
+spec->protocol = g_strdup(qdict_get_try_str(qdict, "protocol"));
+spec->object_name = g_strdup(qdict_get_try_str(qdict, "object-name"));
+
+src = g_strdup(qdict_get_try_str(qdict, "src"));
+if (src) {
+spec->source = g_new0(InetSocketAddressBase, 1);
+
+if (inet_parse_base(spec->source, src, NULL)) {
+monitor_printf(mon, "bad colo passthrough src address");
+goto out;
+}
+}
+
+dst = g_strdup(qdict_get_try_str(qdict, "dst"));
+if (dst) {
+spec->destination = g_new0(InetSocketAddressBase, 1);
+
+if (inet_parse_base(spec->destination, dst, NULL)) {
+monitor_printf(mon, "bad colo passthrough dst address");
+goto out;
+}
+}
+
+qmp_colo_passthrough_add(spec, &err);
+
+out:
+g_free(src);
+src = NULL;
+
+g_free(dst);
+dst = NULL;
+
+hmp_handle_error(mon, err);
+}
+
+void hmp_colo_passthrough_del(Monitor *mon, const QDict *qdict)
+{
+IPFlowSpec *spec = g_new0(IPFlowSpec, 1);
+char *src, *dst;
+Error *err = NULL;
+
+spec->protocol = g_strdup(qdict_get_try_str(qdict, "protocol"));
+spec->object_name = g_strdup(qdict_get_try_str(qdict, "object-name"));
+
+src = g_strdup(qdict_get_try_str(qdict, "src"));
+if (src) {
+spec->source = g_new0(InetSocketAddressBase, 1);
+
+if (inet_parse_base(spec->source, src, NULL)) {
+monitor_printf(mon, "bad colo passthrough src address");
+goto out;
+}
+}
+
+dst = g_strdup(qdict_get_try_str(qdict, "dst"));
+if (dst) {
+spec->destination = g_new0(InetSocketAddressBase, 1);
+
+if (inet_parse_base(spec->destination, dst, NULL)) {
+monitor_printf(mon, "bad colo passthrough dst address");
+goto out;
+}
+}
+
+qmp_colo_passthrough_del(spec, &err);
+
+out:
+g_free(src);
+src = NULL;
+
+g_free(dst);
+dst = NULL;
+
+hmp_handle_error(mon, err);
+}
+
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
 const char *options = qdict_get_str(qdict, "object");
-- 
2.25.1

[PATCH V7 5/6] net/colo-compare: Add passthrough list to CompareState

[Zhang Chen]

Add passthrough list for each CompareState.

Signed-off-by: Zhang Chen 
---
 net/colo-compare.c | 28 
 net/colo-compare.h | 12 
 2 files changed, 40 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 8175612c5f..2250efd40d 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -161,6 +161,7 @@ static int packet_enqueue(CompareState *s, int mode, 
Connection **con)
 ConnectionKey key;
 Packet *pkt = NULL;
 Connection *conn;
+COLOPassthroughEntry *pass, *next;
 int ret;
 
 if (mode == PRIMARY_IN) {
@@ -180,6 +181,31 @@ static int packet_enqueue(CompareState *s, int mode, 
Connection **con)
 }
 fill_connection_key(pkt, &key);
 
+/* Check COLO passthrough specifications */
+qemu_mutex_lock(&s->passthroughlist_mutex);
+if (!QLIST_EMPTY(&s->passthroughlist)) {
+QLIST_FOREACH_SAFE(pass, &s->passthroughlist, node, next) {
+if (key.ip_proto == pass->l4_protocol->p_proto) {
+if (pass->src_port == 0 || pass->src_port == key.dst_port) {
+if (pass->src_ip.s_addr == 0 ||
+pass->src_ip.s_addr == key.src.s_addr) {
+if (pass->dst_port == 0 ||
+pass->dst_port == key.src_port) {
+if (pass->dst_ip.s_addr == 0 ||
+pass->dst_ip.s_addr == key.dst.s_addr) {
+packet_destroy(pkt, NULL);
+pkt = NULL;
+qemu_mutex_unlock(&s->passthroughlist_mutex);
+return -1;
+}
+}
+}
+}
+}
+}
+}
+qemu_mutex_unlock(&s->passthroughlist_mutex);
+
 conn = connection_get(s->connection_track_table,
   &key,
   &s->conn_list);
@@ -1245,6 +1271,7 @@ static void colo_compare_complete(UserCreatable *uc, 
Error **errp)
 }
 
 g_queue_init(&s->conn_list);
+QLIST_INIT(&s->passthroughlist);
 
 s->connection_track_table = g_hash_table_new_full(connection_key_hash,
   connection_key_equal,
@@ -1259,6 +1286,7 @@ static void colo_compare_complete(UserCreatable *uc, 
Error **errp)
 qemu_cond_init(&event_complete_cond);
 colo_compare_active = true;
 }
+qemu_mutex_init(&s->passthroughlist_mutex);
 QTAILQ_INSERT_TAIL(&net_compares, s, next);
 qemu_mutex_unlock(&colo_compare_mutex);
 
diff --git a/net/colo-compare.h b/net/colo-compare.h
index a481df8ce6..1980df4487 100644
--- a/net/colo-compare.h
+++ b/net/colo-compare.h
@@ -23,6 +23,7 @@
 #include "migration/migration.h"
 #include "sysemu/iothread.h"
 #include "colo.h"
+#include 
 
 #define TYPE_COLO_COMPARE "colo-compare"
 typedef struct CompareState CompareState;
@@ -39,6 +40,15 @@ typedef struct COLOSendCo {
 int ret;
 } COLOSendCo;
 
+typedef struct COLOPassthroughEntry {
+struct protoent *l4_protocol;
+int src_port;
+int dst_port;
+struct in_addr src_ip;
+struct in_addr dst_ip;
+QLIST_ENTRY(COLOPassthroughEntry) node;
+} COLOPassthroughEntry;
+
 /*
  *  + CompareState ++
  *  |   |
@@ -95,6 +105,8 @@ struct CompareState {
 
 QEMUBH *event_bh;
 enum colo_event event;
+QLIST_HEAD(, COLOPassthroughEntry) passthroughlist;
+QemuMutex passthroughlist_mutex;
 
 QTAILQ_ENTRY(CompareState) next;
 };
-- 
2.25.1

[PATCH V7 1/6] qapi/net: Add IPFlowSpec and QMP command for COLO passthrough

[Zhang Chen]

Since the real user scenario does not need COLO to monitor all traffic.
Add colo-passthrough-add and colo-passthrough-del to maintain
a COLO network passthrough list. Add IPFlowSpec struct for all QMP commands.
Except protocol field is necessary, other fields are optional.

Signed-off-by: Zhang Chen 
---
 net/net.c | 10 
 qapi/net.json | 68 +++
 2 files changed, 78 insertions(+)

diff --git a/net/net.c b/net/net.c
index 2a472604ec..a4bee86b24 100644
--- a/net/net.c
+++ b/net/net.c
@@ -1194,6 +1194,16 @@ void qmp_netdev_del(const char *id, Error **errp)
 }
 }
 
+void qmp_colo_passthrough_add(IPFlowSpec *spec, Error **errp)
+{
+/* TODO implement setup passthrough rule */
+}
+
+void qmp_colo_passthrough_del(IPFlowSpec *spec, Error **errp)
+{
+/* TODO implement delete passthrough rule */
+}
+
 static void netfilter_print_info(Monitor *mon, NetFilterState *nf)
 {
 char *str;
diff --git a/qapi/net.json b/qapi/net.json
index af3f5b0fda..aca3b54278 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -7,6 +7,7 @@
 ##
 
 { 'include': 'common.json' }
+{ 'include': 'sockets.json' }
 
 ##
 # @set_link:
@@ -694,3 +695,70 @@
 ##
 { 'event': 'FAILOVER_NEGOTIATED',
   'data': {'device-id': 'str'} }
+
+##
+# @IPFlowSpec:
+#
+# IP flow specification.
+#
+# @protocol: Transport layer protocol like TCP/UDP...
+#
+# @object-name: Point out the IPflow spec effective range of object,
+#   If there is no such part, it means global spec.
+#
+# @source: Source address and port.
+#
+# @destination: Destination address and port.
+#
+# Since: 6.1
+##
+{ 'struct': 'IPFlowSpec',
+  'data': { '*protocol': 'str', '*object-name': 'str',
+'*source': 'InetSocketAddressBase',
+'*destination': 'InetSocketAddressBase' } }
+
+##
+# @colo-passthrough-add:
+#
+# Add passthrough entry according to user's needs in COLO-compare.
+# Source IP/port and destination IP/port both optional, If user just
+# input parts of infotmation, it will match all.
+#
+# Returns: Nothing on success
+#
+# Since: 6.1
+#
+# Example:
+#
+# -> { "execute": "colo-passthrough-add",
+#  "arguments": { "protocol": "tcp", "object-name": "object0",
+#  "source": {"host": "192.168.1.1", "port": "1234"},
+#  "destination": {"host": "192.168.1.2", "port": "4321"} } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'colo-passthrough-add', 'boxed': true,
+ 'data': 'IPFlowSpec' }
+
+##
+# @colo-passthrough-del:
+#
+# Delete passthrough entry according to user's needs in COLO-compare.
+# Source IP/port and destination IP/port both optional, If user just
+# input parts of infotmation, it will match all.
+#
+# Returns: Nothing on success
+#
+# Since: 6.1
+#
+# Example:
+#
+# -> { "execute": "colo-passthrough-del",
+#  "arguments": { "protocol": "tcp", "object-name": "object0",
+#  "source": {"host": "192.168.1.1", "port": "1234"},
+#  "destination": {"host": "192.168.1.2", "port": "4321"} } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'colo-passthrough-del', 'boxed': true,
+ 'data': 'IPFlowSpec' }
-- 
2.25.1

[PATCH V7 0/6] Passthrough specific network traffic in COLO

[Zhang Chen]

Due to some real user scenarios don't need to monitor all traffic.
And qemu net-filter also need function to more detailed flow control.
This series give user ability to passthrough kinds of COLO network stream.

For example, windows guest user want to enable windows remote desktop
to touch guest(UDP/TCP 3389), This case use UDP and TCP mixed, and the
tcp part payload always different caused by real desktop display
data(for guest time/ mouse display).

Another case is some real user application will actively transmit information
include guest time part, primary guest send data with time 10:01.000,
At the same time secondary guest send data with time 10:01.001,
it will always trigger COLO checkpoint(live migrate) to drop guest performance.

  V7:
- Keep some data structure stay in .c (patch 4/6).
- Fix mutex init issue (patch 5/6).
- Make the IPFlowSpec 'protocol' field optional (patch 1/6).
- Add compare_passthrough_find function in net.c (patch 6/6).

  V6:
- Change QAPI IPFlowSpec protocol from enum to str.
- Use getprotobyname to handle the protocols.
- Optimize code in net.

  V5:
- Squash original 1-3 QAPI patches together.
- Rename some data structures to avoid misunderstanding.
- Reuse InetSocketAddressBase in IPFlowSpec.
- Add new function in util/qemu-sockets.c to parse
  InetSocketAddressBase.
- Update HMP command define to reuse current code.
- Add more comments.

  V4:
- Fix QAPI code conflict for V6.0 merged patches.
- Note this feature for V6.1.

  V3:
- Add COLO passthrough list lock.
- Add usage demo and more comments.

  V2:
- Add the n-tuple support.
- Add some qapi definitions.
- Support multi colo-compare objects.
- Support setup each rules for each objects individually.
- Clean up COLO compare definition to .h file.
- Rebase HMP command for stable tree.
- Add redundant rules check.


Zhang Chen (6):
  qapi/net: Add IPFlowSpec and QMP command for COLO passthrough
  util/qemu-sockets.c: Add inet_parse_base to handle
InetSocketAddressBase
  hmp-commands: Add new HMP command for COLO passthrough
  net/colo-compare: Move data structure and define to .h file.
  net/colo-compare: Add passthrough list to CompareState
  net/net.c: Add handler for COLO passthrough connection

 hmp-commands.hx|  26 +++
 include/monitor/hmp.h  |   2 +
 include/qemu/sockets.h |   1 +
 monitor/hmp-cmds.c |  82 
 net/colo-compare.c | 160 +--
 net/colo-compare.h |  98 
 net/net.c  | 168 +
 qapi/net.json  |  68 +
 util/qemu-sockets.c|  14 
 9 files changed, 510 insertions(+), 109 deletions(-)

-- 
2.25.1

Re: proposed schedule for 6.1 release

[Richard Henderson]

On 5/24/21 5:27 AM, Peter Maydell wrote:

Here's a draft schedule for the 6.1 cycle:

2021-07-13 Soft feature freeze. Only bug fixes after this point.
2021-07-20 Hard feature freeze. Tag rc0
2021-07-27 Tag rc1
2021-08-03 Tag rc2
2021-08-10 Tag rc3
2021-08-17 Release; or tag rc4 if needed
2021-08-24 Release if we needed an rc4

I don't think there's anything much we want to try to avoid
dates-wise, so I just put the target release date in the middle
of August and worked back from there.

Any objections/suggestions ?


LGTM.


r~

Re: [PATCH v7 13/23] cpu: Move AVR target vmsd field from CPUClass to DeviceClass

[Richard Henderson]

On 5/17/21 3:51 AM, Philippe Mathieu-Daudé wrote:

See rationale in previous commit. Targets should use the vmsd field
of DeviceClass, not CPUClass. As migration is not important on the
AVR target, break the migration compatibility and set the DeviceClass
vmsd field. To feel safer, increment the vmstate version.

Signed-off-by: Philippe Mathieu-Daudé
---
v7: Increment vmstate version (Richard)
---
  target/avr/cpu.c | 2 +-
  target/avr/machine.c | 4 ++--
  2 files changed, 3 insertions(+), 3 deletions(-)


Reviewed-by: Richard Henderson 

r~

Re: [PATCH v7 05/23] cpu: Split as cpu-common / cpu-sysemu

[Richard Henderson]

On 5/17/21 3:51 AM, Philippe Mathieu-Daudé wrote:

The current cpu.c contains sysemu-specific methods.
To avoid building them in user-mode builds, split the
current cpu.c as cpu-common.c / cpu-sysemu.c.

Start by moving cpu_get_crash_info().

Signed-off-by: Philippe Mathieu-Daudé
---
  hw/core/{cpu.c => cpu-common.c} | 17 -
  hw/core/cpu-sysemu.c| 34 +
  hw/core/meson.build |  3 ++-
  3 files changed, 36 insertions(+), 18 deletions(-)
  rename hw/core/{cpu.c => cpu-common.c} (96%)
  create mode 100644 hw/core/cpu-sysemu.c


Reviewed-by: Richard Henderson 

r~

Re: [PATCH v7 02/23] cpu: Restrict target cpu_do_transaction_failed() handlers to sysemu

[Richard Henderson]

On 5/17/21 3:51 AM, Philippe Mathieu-Daudé wrote:

In commit cbc183d2d9f ("cpu: move cc->transaction_failed to tcg_ops")
we restricted the do_transaction_failed() handler to the sysemu part
of TCGCPUOps, but forgot to restrict the target specific declarations.

Signed-off-by: Philippe Mathieu-Daudé
---
  target/arm/internals.h |  2 ++
  target/m68k/cpu.h  |  2 ++
  target/riscv/cpu.h | 10 +-
  target/xtensa/cpu.h|  8 
  4 files changed, 13 insertions(+), 9 deletions(-)


What do the extra ifdefs buy us? Surely the fact that the symbol is not present 
in the user-only, and would produce link errors if used, is sufficient?


r~

Re: [PATCH v7 04/23] cpu: Remove duplicated 'sysemu/hw_accel.h' header

[Richard Henderson]

On 5/17/21 3:51 AM, Philippe Mathieu-Daudé wrote:

Signed-off-by: Philippe Mathieu-Daudé
---
  hw/core/cpu.c | 1 -
  1 file changed, 1 deletion(-)


Reviewed-by: Richard Henderson 

r~

Re: [PATCH 2/5] linux-uesr: make exec_path realpath

[Takashi Yamamoto]

On Tue, May 25, 2021 at 7:59 AM Takashi Yamamoto  wrote:
>
> On Mon, May 24, 2021 at 7:59 PM Alex Bennée  wrote:
> >
> >
> > YAMAMOTO Takashi  writes:
> >
> > > Otherwise, it can be easily fooled by the user app using chdir().
> > >
> > > Signed-off-by: YAMAMOTO Takashi 
> > > ---
> > >  linux-user/main.c | 6 +-
> > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/linux-user/main.c b/linux-user/main.c
> > > index 4dfc47ad3b..1f9f4e3820 100644
> > > --- a/linux-user/main.c
> > > +++ b/linux-user/main.c
> > > @@ -55,6 +55,7 @@
> > >  #endif
> > >
> > >  char *exec_path;
> > > +char exec_path_store[PATH_MAX];
> >
> > Is there any point in keeping this as a static path rather than just
> > allocating it off the heap?
>
> it's just the simplest given the api of realpath().
> do you mean it's better to use malloc? why?
>
> >
> > >
> > >  int singlestep;
> > >  static const char *argv0;
> > > @@ -610,7 +611,10 @@ static int parse_args(int argc, char **argv)
> > >  exit(EXIT_FAILURE);
> > >  }
> > >
> > > -exec_path = argv[optind];
> > > +exec_path = realpath(argv[optind], exec_path_store);
> > > +if (exec_path == NULL) {
> > > +exec_path = argv[optind];
> > > +}
> >
> >   exec_path = realpath(argv[optind], NULL)
> >   exec_path = exec_path ? exec_path : argv[optind];
> >
> > what situations would we expect realpath to fail and in those cases is
> > sticking to argv[optind] safe?
>
> i don't have any particular case in my mind.
> i guess it rarely fails and it might be simpler to just bail out on the 
> failure.

i recalled why i did this way.
it was actually necessary for some apps. eg. runc
it executes an unlinked binary via /proc/self/fd.
i'll clean it up a bit and add comments.

>
> >
> > >
> > >  return optind;
> > >  }
> >
> >
> > --
> > Alex Bennée

Re: [RFC v3 13/29] vhost: Add vhost_get_iova_range operation

[Jason Wang]

在 2021/5/20 上午12:28, Eugenio Pérez 写道:

For simplicity, If a device does not support this operation it means
that it can handle full (uint64_t)-1 iova address.



Note that, we probably need a separated patch for this.

And we need to this during vhost-vdpa initialization. If GPA is out of 
the range, we need to fail the start of vhost-vdpa.


THanks




Signed-off-by: Eugenio Pérez 
---
  include/hw/virtio/vhost-backend.h |  5 +
  hw/virtio/vhost-vdpa.c| 18 ++
  hw/virtio/trace-events|  1 +
  3 files changed, 24 insertions(+)

diff --git a/include/hw/virtio/vhost-backend.h 
b/include/hw/virtio/vhost-backend.h
index 94d3323905..bcb112c166 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -36,6 +36,7 @@ struct vhost_vring_addr;
  struct vhost_scsi_target;
  struct vhost_iotlb_msg;
  struct vhost_virtqueue;
+struct vhost_vdpa_iova_range;
  
  typedef int (*vhost_backend_init)(struct vhost_dev *dev, void *opaque);

  typedef int (*vhost_backend_cleanup)(struct vhost_dev *dev);
@@ -127,6 +128,9 @@ typedef bool (*vhost_force_iommu_op)(struct vhost_dev *dev);
  
  typedef int (*vhost_vring_pause_op)(struct vhost_dev *dev);
  
+typedef int (*vhost_get_iova_range)(struct vhost_dev *dev,

+hwaddr *first, hwaddr *last);
+
  typedef struct VhostOps {
  VhostBackendType backend_type;
  vhost_backend_init vhost_backend_init;
@@ -173,6 +177,7 @@ typedef struct VhostOps {
  vhost_get_device_id_op vhost_get_device_id;
  vhost_vring_pause_op vhost_vring_pause;
  vhost_force_iommu_op vhost_force_iommu;
+vhost_get_iova_range vhost_get_iova_range;
  } VhostOps;
  
  extern const VhostOps user_ops;

diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 01d2101d09..74fe92935e 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -579,6 +579,23 @@ static bool  vhost_vdpa_force_iommu(struct vhost_dev *dev)
  return true;
  }
  
+static int vhost_vdpa_get_iova_range(struct vhost_dev *dev,

+ hwaddr *first, hwaddr *last)
+{
+int ret;
+struct vhost_vdpa_iova_range range;
+
+ret = vhost_vdpa_call(dev, VHOST_VDPA_GET_IOVA_RANGE, &range);
+if (ret != 0) {
+return ret;
+}
+
+*first = range.first;
+*last = range.last;
+trace_vhost_vdpa_get_iova_range(dev, *first, *last);
+return ret;
+}
+
  const VhostOps vdpa_ops = {
  .backend_type = VHOST_BACKEND_TYPE_VDPA,
  .vhost_backend_init = vhost_vdpa_init,
@@ -611,4 +628,5 @@ const VhostOps vdpa_ops = {
  .vhost_get_device_id = vhost_vdpa_get_device_id,
  .vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
  .vhost_force_iommu = vhost_vdpa_force_iommu,
+.vhost_get_iova_range = vhost_vdpa_get_iova_range,
  };
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index c62727f879..5debe3a681 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) 
"dev: %p index:
  vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 
0x%"PRIx64
  vhost_vdpa_set_owner(void *dev) "dev: %p"
  vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t 
used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 
0x%"PRIx64" used_user_addr: 0x%"PRIx64
+vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 
0x%"PRIx64" last: 0x%"PRIx64
  
  # virtio.c

  virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) 
"elem %p size %zd in_num %u out_num %u"

[RFC PATCH] linux-user/sparc: Implement v8plus signals

[Richard Henderson]

Sparc v8plus is a sparc64 running a 32-bit ABI.
The significant difference vs sparc32 is that all 64 bits of
the %g and %o registers, plus %xcc, are saved across interrupts,
context switches, and signals.

There's a special marker in the saved %psr value that's used to
indicate that %xcc and the high bits are present in the frame.

Signed-off-by: Richard Henderson 
---

I have been unable to find an extant v8plus distribution with
which to test this beyond compilation.  Thus the RFC.  I know
debian used to have one, but they have moved to pure sparc64 now.

Thoughts?


r~

---
 target/sparc/cpu.h|  2 -
 linux-user/sparc/signal.c | 96 ---
 2 files changed, 80 insertions(+), 18 deletions(-)

diff --git a/target/sparc/cpu.h b/target/sparc/cpu.h
index ff8ae73002..d2c6e2e4ec 100644
--- a/target/sparc/cpu.h
+++ b/target/sparc/cpu.h
@@ -104,14 +104,12 @@ enum {
 #define PSR_CARRY_SHIFT 20
 #define PSR_CARRY (1 << PSR_CARRY_SHIFT)
 #define PSR_ICC   (PSR_NEG|PSR_ZERO|PSR_OVF|PSR_CARRY)
-#if !defined(TARGET_SPARC64)
 #define PSR_EF(1<<12)
 #define PSR_PIL   0xf00
 #define PSR_S (1<<7)
 #define PSR_PS(1<<6)
 #define PSR_ET(1<<5)
 #define PSR_CWP   0x1f
-#endif
 
 #define CC_SRC (env->cc_src)
 #define CC_SRC2 (env->cc_src2)
diff --git a/linux-user/sparc/signal.c b/linux-user/sparc/signal.c
index 0cc3db5570..d3d699545b 100644
--- a/linux-user/sparc/signal.c
+++ b/linux-user/sparc/signal.c
@@ -61,6 +61,13 @@ struct target_siginfo_fpu {
 #endif
 };
 
+struct target_siginfo_v8plus {
+#if defined(TARGET_SPARC64) && defined(TARGET_ABI32)
+uint32_t u_regs_upper[16];
+uint32_t asi;
+#endif
+};
+
 #ifdef TARGET_ARCH_HAS_SETUP_FRAME
 struct target_signal_frame {
 struct target_stackf ss;
@@ -69,7 +76,8 @@ struct target_signal_frame {
 abi_ulong fpu_save;
 uint32_t insns[2] QEMU_ALIGNED(8);
 abi_ulong extramask[TARGET_NSIG_WORDS - 1];
-abi_ulong extra_size; /* Should be 0 */
+abi_ulong extra_size; /* Should be sizeof(v8plus) */
+struct target_siginfo_v8plus v8plus;
 abi_ulong rwin_save;
 };
 #endif
@@ -87,8 +95,9 @@ struct target_rt_signal_frame {
 abi_ulong fpu_save;
 uint32_t insns[2];
 target_stack_t stack;
-abi_ulong extra_size; /* Should be 0 */
+abi_ulong extra_size; /* Should be sizeof(v8plus) */
 #endif
+struct target_siginfo_v8plus v8plus;
 abi_ulong rwin_save;
 };
 
@@ -121,7 +130,34 @@ static abi_ulong get_sigframe(struct target_sigaction *sa,
 return sp;
 }
 
-static void save_pt_regs(struct target_pt_regs *regs, CPUSPARCState *env)
+/* Fake PSR_IMPL | PSR_VER, meaning 64-bit cpu is present. */
+#define PSR_V8PLUS  0xff00u
+/* If PSR_V8PLUS, this field contains %xcc. */
+#define PSR_XCC 0x000fu
+
+#if defined(TARGET_SPARC64) && defined(TARGET_ABI32)
+# define SAVE_REG(X, I) \
+do {\
+__put_user(X, ®s->u_regs[I]);\
+__put_user(X >> 32, &v8plus->u_regs_upper[I]);  \
+} while (0)
+# define RESTORE_REG(X, I)  \
+do {\
+uint32_t l_, h_ = 0;\
+__get_user(l_, ®s->u_regs[I]);   \
+if ((psr & PSR_V8PLUS) == PSR_V8PLUS) { \
+__get_user(h_, &v8plus->u_regs_upper[I]);   \
+}   \
+X = deposit64(l_, 32, 32, h_);  \
+} while (0)
+#else
+# define SAVE_REG(X, I) __put_user(X, ®s->u_regs[I])
+# define RESTORE_REG(X, I)  __get_user(X, ®s->u_regs[I])
+#endif
+
+static void save_pt_regs(struct target_pt_regs *regs,
+ struct target_siginfo_v8plus *v8plus,
+ CPUSPARCState *env)
 {
 int i;
 
@@ -130,7 +166,18 @@ static void save_pt_regs(struct target_pt_regs *regs, 
CPUSPARCState *env)
 /* TODO: magic should contain PT_REG_MAGIC + %tt. */
 __put_user(0, ®s->magic);
 #else
-__put_user(cpu_get_psr(env), ®s->psr);
+uint32_t psr;
+# ifdef TARGET_SPARC64
+/* See linux/arch/sparc/include/uapi/asm/psrcompat.h, tstate_to_psr */
+uint64_t tstate = sparc64_tstate(env);
+psr = (tstate & 0x1f) /* TSTATE_CWP */
+| ((tstate >> 12) & PSR_ICC)
+| ((tstate >> 20) & PSR_XCC)
+| PSR_S | PSR_V8PLUS;
+# else
+psr = cpu_get_psr(env);
+# endif
+__put_user(psr, ®s->psr);
 #endif
 
 __put_user(env->pc, ®s->pc);
@@ -138,14 +185,20 @@ static void save_pt_regs(struct target_pt_regs *regs, 
CPUSPARCState *env)
 __put_user(env->y, ®s->y);
 
 for (i = 0; i < 8; i++) {
-__put_user(env->gregs[i], ®s->u_regs[i]);
+SAVE_REG(env->gregs[i], i);
 }
 for (i = 0; i < 8; i++) {
-__put_user(env->regwptr[WREG_O0 + i], ®s->u_regs[i + 8]);
+SAVE_REG(env->regwptr[WREG_O0 + i], i + 8);
 }
+
+#if defined(TARGET_SPARC64)

Re: [RFC v3 06/29] virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED

[Jason Wang]

在 2021/5/26 上午9:06, Jason Wang 写道:


在 2021/5/20 上午12:28, Eugenio Pérez 写道:

So the guest can stop and start net device. It implements the RFC
https://lists.oasis-open.org/archives/virtio-comment/202012/msg00027.html 



To stop (as "pause") the device is required to migrate status and vring
addresses between device and SVQ.

This is a WIP commit: as with VIRTIO_F_QUEUE_STATE, is introduced in
virtio_config.h before of even proposing for the kernel, with no feature
flag, and, with no checking in the device. It also needs a modified
vp_vdpa driver that supports to set and retrieve status.

Signed-off-by: Eugenio Pérez 
---
  include/standard-headers/linux/virtio_config.h | 2 ++
  hw/net/virtio-net.c    | 4 +++-
  2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/standard-headers/linux/virtio_config.h 
b/include/standard-headers/linux/virtio_config.h

index 59fad3eb45..b3f6b1365d 100644
--- a/include/standard-headers/linux/virtio_config.h
+++ b/include/standard-headers/linux/virtio_config.h
@@ -40,6 +40,8 @@
  #define VIRTIO_CONFIG_S_DRIVER_OK    4
  /* Driver has finished configuring features */
  #define VIRTIO_CONFIG_S_FEATURES_OK    8
+/* Device is stopped */
+#define VIRTIO_CONFIG_S_DEVICE_STOPPED 32
  /* Device entered invalid state, driver must reset it */
  #define VIRTIO_CONFIG_S_NEEDS_RESET    0x40
  /* We've given up on this device. */
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 96a3cc8357..2d3caea289 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -198,7 +198,9 @@ static bool virtio_net_started(VirtIONet *n, 
uint8_t status)

  {
  VirtIODevice *vdev = VIRTIO_DEVICE(n);
  return (status & VIRTIO_CONFIG_S_DRIVER_OK) &&
-    (n->status & VIRTIO_NET_S_LINK_UP) && vdev->vm_running;
+    (!(n->status & VIRTIO_CONFIG_S_DEVICE_STOPPED)) &&
+    (n->status & VIRTIO_NET_S_LINK_UP) &&
+    vdev->vm_running;
  }
    static void virtio_net_announce_notify(VirtIONet *net)



It looks to me this is only the part of pause. 



And even for pause, I don't see anything that prevents rx/tx from being 
executed? (E.g virtio_net_handle_tx_bh or virtio_net_handle_rx).


Thanks



We still need the resume?

Thanks

Re: [RFC v3 06/29] virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED

[Jason Wang]

在 2021/5/20 上午12:28, Eugenio Pérez 写道:

So the guest can stop and start net device. It implements the RFC
https://lists.oasis-open.org/archives/virtio-comment/202012/msg00027.html

To stop (as "pause") the device is required to migrate status and vring
addresses between device and SVQ.

This is a WIP commit: as with VIRTIO_F_QUEUE_STATE, is introduced in
virtio_config.h before of even proposing for the kernel, with no feature
flag, and, with no checking in the device. It also needs a modified
vp_vdpa driver that supports to set and retrieve status.

Signed-off-by: Eugenio Pérez 
---
  include/standard-headers/linux/virtio_config.h | 2 ++
  hw/net/virtio-net.c| 4 +++-
  2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/standard-headers/linux/virtio_config.h 
b/include/standard-headers/linux/virtio_config.h
index 59fad3eb45..b3f6b1365d 100644
--- a/include/standard-headers/linux/virtio_config.h
+++ b/include/standard-headers/linux/virtio_config.h
@@ -40,6 +40,8 @@
  #define VIRTIO_CONFIG_S_DRIVER_OK 4
  /* Driver has finished configuring features */
  #define VIRTIO_CONFIG_S_FEATURES_OK   8
+/* Device is stopped */
+#define VIRTIO_CONFIG_S_DEVICE_STOPPED 32
  /* Device entered invalid state, driver must reset it */
  #define VIRTIO_CONFIG_S_NEEDS_RESET   0x40
  /* We've given up on this device. */
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 96a3cc8357..2d3caea289 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -198,7 +198,9 @@ static bool virtio_net_started(VirtIONet *n, uint8_t status)
  {
  VirtIODevice *vdev = VIRTIO_DEVICE(n);
  return (status & VIRTIO_CONFIG_S_DRIVER_OK) &&
-(n->status & VIRTIO_NET_S_LINK_UP) && vdev->vm_running;
+(!(n->status & VIRTIO_CONFIG_S_DEVICE_STOPPED)) &&
+(n->status & VIRTIO_NET_S_LINK_UP) &&
+vdev->vm_running;
  }
  
  static void virtio_net_announce_notify(VirtIONet *net)



It looks to me this is only the part of pause. We still need the resume?

Thanks

[PATCH] .mailmap: Update my email address

[Finn Thain]

Signed-off-by: Finn Thain 
---
 .mailmap | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.mailmap b/.mailmap
index a1bd659817..772f3e16cf 100644
--- a/.mailmap
+++ b/.mailmap
@@ -91,6 +91,7 @@ Erik Smit 
 Fabrice Desclaux 
 Fernando Luis Vázquez Cao 
 Fernando Luis Vázquez Cao 
+Finn Thain  
 Gautham R. Shenoy 
 Gautham R. Shenoy 
 Gonglei (Arei) 
-- 
2.26.3

Re: [PATCH 5/5] linux-user: Implement pivot_root

[Takashi Yamamoto]

On Wed, May 26, 2021 at 5:22 AM Laurent Vivier  wrote:
>
> Le 24/05/2021 à 06:54, YAMAMOTO Takashi a écrit :
> > Used by runc.
> >
> > Signed-off-by: YAMAMOTO Takashi 
> > ---
> >  linux-user/syscall.c | 23 +++
> >  1 file changed, 23 insertions(+)
> >
> > diff --git a/linux-user/syscall.c b/linux-user/syscall.c
> > index 2947e79dc0..e739921e86 100644
> > --- a/linux-user/syscall.c
> > +++ b/linux-user/syscall.c
> > @@ -35,6 +35,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
>
> we don't need that include, see below
>
> >  #include 
> >  #include 
> >  #include 
> > @@ -8254,6 +8255,11 @@ static int host_to_target_cpu_mask(const unsigned 
> > long *host_mask,
> >  return 0;
> >  }
> >
> > +static int pivot_root(const char *new_root, const char *put_old)
> > +{
> > +return syscall(SYS_pivot_root, new_root, put_old);
> > +}
>
> Better to use:
>
> #if defined(TARGET_NR_pivot_root) && defined(__NR_pivot_root)
> _syscall2(int, pivot_root, const char *, new_root, const char *, put_old)
> #endif

ok. i haven't noticed the _syscall2 macro in this file. thank you.

>
> > +
> >  /* This is an internal helper for do_syscall so that it is easier
> >   * to have a single return point, so that actions, such as logging
> >   * of syscall results, can be performed.
> > @@ -13222,6 +13228,23 @@ static abi_long do_syscall1(void *cpu_env, int 
> > num, abi_long arg1,
> >  return ret;
> >  #endif
> >
> > +#if defined(TARGET_NR_pivot_root)
> > +case TARGET_NR_pivot_root:
> > +{
> > +void *p2;
> > +p = lock_user_string(arg1); /* new_root */
> > +p2 = lock_user_string(arg2); /* put_old */
> > +if (!p || !p2) {
> > +ret = -TARGET_EFAULT;
> > +} else {
> > +ret = get_errno(pivot_root(p, p2));
> > +}
> > +unlock_user(p2, arg2, 0);
> > +unlock_user(p, arg1, 0);
> > +}
> > +return ret;
> > +#endif
> > +
> >  default:
> >  qemu_log_mask(LOG_UNIMP, "Unsupported syscall: %d\n", num);
> >  return -TARGET_ENOSYS;
> >
>
> Thanks,
> Laurent
>

[PATCH v7 29/31] python: add .gitignore

[John Snow]

Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build. They are created by
running the commands 'python3 setup.py ' when preparing
tarballs to upload to e.g. PyPI.

Ignore miscellaneous cached python confetti (mypy, pylint, et al)

Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).

Signed-off-by: John Snow 
---
 python/.gitignore | 15 +++
 1 file changed, 15 insertions(+)
 create mode 100644 python/.gitignore

diff --git a/python/.gitignore b/python/.gitignore
new file mode 100644
index 000..4ed144ceac3
--- /dev/null
+++ b/python/.gitignore
@@ -0,0 +1,15 @@
+# linter/tooling cache
+.mypy_cache/
+.cache/
+
+# python packaging
+build/
+dist/
+qemu.egg-info/
+
+# editor config
+.idea/
+.vscode/
+
+# virtual environments (pipenv et al)
+.venv/
-- 
2.31.1

[PATCH v7 30/31] python: add tox support

[John Snow]

This is intended to be a manually run, non-CI script.

Use tox to test the linters against all python versions from 3.6 to
3.10. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:

> sudo dnf install python3.6 python3.7 python3.8 python3.9 python3.10

Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.

With confidence that the tests pass on 3.6 through 3.10 inclusive, add
the appropriate classifiers to setup.cfg to indicate which versions we
claim to support.

Tox 3.18.0 or above is required to use the 'allowlist_externals' option.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/.gitignore |  1 +
 python/Makefile   |  7 ++-
 python/setup.cfg  | 23 ++-
 3 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/python/.gitignore b/python/.gitignore
index 4ed144ceac3..272ed223a84 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -13,3 +13,4 @@ qemu.egg-info/
 
 # virtual environments (pipenv et al)
 .venv/
+.tox/
diff --git a/python/Makefile b/python/Makefile
index a9da1689558..b5621b0d540 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -16,6 +16,8 @@ help:
@echo ""
@echo "make check:  run linters using the current environment."
@echo ""
+   @echo "make check-tox:  run linters using multiple python versions."
+   @echo ""
@echo "make clean:  remove package build output."
@echo ""
@echo "make distclean:  remove venv files, qemu package forwarder,"
@@ -36,8 +38,11 @@ develop:
 check:
@avocado --config avocado.cfg run tests/
 
+check-tox:
+   @tox
+
 clean:
python3 setup.py clean --all
 
 distclean: clean
-   rm -rf qemu.egg-info/ .venv/ dist/
+   rm -rf qemu.egg-info/ .venv/ .tox/ dist/
diff --git a/python/setup.cfg b/python/setup.cfg
index fd325194901..0fcdec6f322 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -14,6 +14,11 @@ classifiers =
 Natural Language :: English
 Operating System :: OS Independent
 Programming Language :: Python :: 3 :: Only
+Programming Language :: Python :: 3.6
+Programming Language :: Python :: 3.7
+Programming Language :: Python :: 3.8
+Programming Language :: Python :: 3.9
+Programming Language :: Python :: 3.10
 
 [options]
 python_requires = >= 3.6
@@ -30,12 +35,13 @@ devel =
 isort >= 5.1.2
 mypy >= 0.770
 pylint >= 2.8.0
-
+tox >= 3.18.0
 
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 exclude = __pycache__,
   .venv,
+  .tox,
 
 [mypy]
 strict = True
@@ -79,3 +85,18 @@ include_trailing_comma=True
 line_length=72
 lines_after_imports=2
 multi_line_output=3
+
+# tox (https://tox.readthedocs.io/) is a tool for running tests in
+# multiple virtualenvs. This configuration file will run the test suite
+# on all supported python versions. To use it, "pip install tox" and
+# then run "tox" from this directory. You will need all of these versions
+# of python available on your system to run this test.
+
+[tox:tox]
+envlist = py36, py37, py38, py39, py310
+
+[testenv]
+allowlist_externals = make
+deps = .[devel]
+commands =
+make check
-- 
2.31.1

[PATCH v7 28/31] python: add Makefile for some common tasks

[John Snow]

Add "make venv" to create the pipenv-managed virtual environment that
contains our explicitly pinned dependencies.

Add "make check" to run the python linters [in the host execution
environment].

Add "make venv-check" which combines the above two: create/update the
venv, then run the linters in that explicitly managed environment.

Add "make develop" which canonizes the runes needed to get both the
linting pre-requisites (the "[devel]" part), and the editable
live-install (the "-e" part) of these python libraries.

make clean: delete miscellaneous python packaging output possibly
created by pipenv, pip, or other python packaging utilities

make distclean: delete the above, the .venv, and the editable "qemu"
package forwarder (qemu.egg-info) if there is one.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/PACKAGE.rst |  6 ++
 python/README.rst  |  6 ++
 python/Makefile| 43 +++
 3 files changed, 55 insertions(+)
 create mode 100644 python/Makefile

diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
index 05ea7789fc1..b0b86cc4c31 100644
--- a/python/PACKAGE.rst
+++ b/python/PACKAGE.rst
@@ -35,3 +35,9 @@ the report.
 Optional packages necessary for running code quality analysis for this
 package can be installed with the optional dependency group "devel":
 ``pip install qemu[devel]``.
+
+``make develop`` can be used to install this package in editable mode
+(to the current environment) *and* bring in testing dependencies in one
+command.
+
+``make check`` can be used to run the available tests.
diff --git a/python/README.rst b/python/README.rst
index 6bd2c6b3547..dcf993819db 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -28,6 +28,9 @@ Installing ".[devel]" instead of "." will additionally pull 
in required
 packages for testing this package. They are not runtime requirements,
 and are not needed to simply use these libraries.
 
+Running ``make develop`` will pull in all testing dependencies and
+install QEMU in editable mode to the current environment.
+
 See `Installing packages using pip and virtual environments
 
`_
 for more information.
@@ -39,6 +42,9 @@ Files in this directory
 - ``qemu/`` Python package source directory.
 - ``tests/`` Python package tests directory.
 - ``avocado.cfg`` Configuration for the Avocado test-runner.
+  Used by ``make check`` et al.
+- ``Makefile`` provides some common testing/installation invocations.
+  Try ``make help`` to see available targets.
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
diff --git a/python/Makefile b/python/Makefile
new file mode 100644
index 000..a9da1689558
--- /dev/null
+++ b/python/Makefile
@@ -0,0 +1,43 @@
+.PHONY: help venv venv-check check clean distclean develop
+
+help:
+   @echo "python packaging help:"
+   @echo ""
+   @echo "make venv:   Create pipenv's virtual environment."
+   @echo "NOTE: Requires Python 3.6 and pipenv."
+   @echo "  Will download packages from PyPI."
+   @echo "Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
+   @echo ""
+   @echo "make venv-check: run linters using pipenv's virtual environment."
+   @echo "Hint: If you don't know which test to run, run this one!"
+   @echo ""
+   @echo "make develop:Install deps for 'make check', and"
+   @echo " the qemu libs in editable/development mode."
+   @echo ""
+   @echo "make check:  run linters using the current environment."
+   @echo ""
+   @echo "make clean:  remove package build output."
+   @echo ""
+   @echo "make distclean:  remove venv files, qemu package forwarder,"
+   @echo " built distribution files, and everything"
+   @echo " from 'make clean'."
+
+venv: .venv
+.venv: Pipfile.lock
+   @PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
+   @touch .venv
+
+venv-check: venv
+   @pipenv run make check
+
+develop:
+   pip3 install -e .[devel]
+
+check:
+   @avocado --config avocado.cfg run tests/
+
+clean:
+   python3 setup.py clean --all
+
+distclean: clean
+   rm -rf qemu.egg-info/ .venv/ dist/
-- 
2.31.1

[PATCH v7 26/31] python: add devel package requirements to setuptools

[John Snow]

setuptools doesn't have a formal understanding of development requires,
but it has an optional feataures section. Fine; add a "devel" feature
and add the requirements to it.

To avoid duplication, we can modify pipenv to install qemu[devel]
instead. This enables us to run invocations like "pip install -e
.[devel]" and test the package on bleeding-edge packages beyond those
specified in Pipfile.lock.

Importantly, this also allows us to install the qemu development
packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
will now fail if the proper development dependencies are not already
met. This can be useful for automated build scripts where fetching
network packages may be undesirable.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/PACKAGE.rst  |  4 
 python/README.rst   |  4 
 python/Pipfile  |  5 +
 python/Pipfile.lock | 14 +-
 python/setup.cfg|  9 +
 5 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
index 1bbfe1b58e2..05ea7789fc1 100644
--- a/python/PACKAGE.rst
+++ b/python/PACKAGE.rst
@@ -31,3 +31,7 @@ official `GitLab mirror 
`_.
 Please report bugs on the `QEMU issue tracker
 `_ and tag ``@jsnow`` in
 the report.
+
+Optional packages necessary for running code quality analysis for this
+package can be installed with the optional dependency group "devel":
+``pip install qemu[devel]``.
diff --git a/python/README.rst b/python/README.rst
index bf9bbca979a..954870973d0 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -24,6 +24,10 @@ which installs a version of the package that installs a 
forwarder
 pointing to these files, such that the package always reflects the
 latest version in your git tree.
 
+Installing ".[devel]" instead of "." will additionally pull in required
+packages for testing this package. They are not runtime requirements,
+and are not needed to simply use these libraries.
+
 See `Installing packages using pip and virtual environments
 
`_
 for more information.
diff --git a/python/Pipfile b/python/Pipfile
index dbe96f71c48..e7acb8cefa4 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,10 +4,7 @@ url = "https://pypi.org/simple";
 verify_ssl = true
 
 [dev-packages]
-flake8 = ">=3.6.0"
-isort = ">=5.1.2"
-mypy = ">=0.770"
-pylint = ">=2.8.0"
+qemu = {editable = true, extras = ["devel"], path = "."}
 
 [packages]
 qemu = {editable = true,path = "."}
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index f0bf576c31e..a2cdc1c50ea 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
 "_meta": {
 "hash": {
-"sha256": 
"7c74cc4c2db3a75c954a6686411cda6fd60e464620bb6d5f1ed9a54be61db4cc"
+"sha256": 
"eff562a688ebc6f3ffe67494dbb804b883e2159ad81c4d55d96da9f7aec13e91"
 },
 "pipfile-spec": 6,
 "requires": {
@@ -35,7 +35,7 @@
 
"sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b",
 
"sha256:bf8fd46d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907"
 ],
-"index": "pypi",
+"markers": "python_version >= '2.7' and python_version not in 
'3.0, 3.1, 3.2, 3.3, 3.4'",
 "version": "==3.9.2"
 },
 "importlib-metadata": {
@@ -51,7 +51,7 @@
 
"sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
 
"sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
 ],
-"index": "pypi",
+"markers": "python_version >= '3.6' and python_version < '4.0'",
 "version": "==5.8.0"
 },
 "lazy-object-proxy": {
@@ -114,7 +114,7 @@
 
"sha256:d65cc1df038ef55a99e617431f0553cd77763869eebdf9042403e16089fe746c",
 
"sha256:d7da2e1d5f558c37d6e8c1246f1aec1e7349e4913d8fb3cb289a35de573fe2eb"
 ],
-"index": "pypi",
+"markers": "python_version >= '3.5'",
 "version": "==0.812"
 },
 "mypy-extensions": {
@@ -145,9 +145,13 @@
 
"sha256:586d8fa9b1891f4b725f587ef267abe2a1bad89d6b184520c7f07a253dd6e217",
 
"sha256:f7e2072654a6b6afdf5e2fb38147d3e2d2d43c89f648637baab63e026481279b"
 ],
-"index": "pypi",
+"markers": "python_version ~= '3.6'",
 "version": "==2.8.2"
 },
+"qemu": {
+"editable": true,
+"path": "."
+},
 "toml": {
 "hashes": [
 
"sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
diff --git a/python/setup.cfg b/python/setup.cfg
index 3f07bd2752d..39dc135e601 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -22,6 +22

[PATCH v7 27/31] python: add avocado-framework and tests

[John Snow]

Try using avocado to manage our various tests; even though right now
they're only invoking shell scripts and not really running any
python-native code.

Create tests/, and add shell scripts which call out to mypy, flake8,
pylint and isort to enforce the standards in this directory.

Add avocado-framework to the setup.cfg development dependencies, and add
avocado.cfg to store some preferences for how we'd like the test output
to look.

Finally, add avocado-framework to the Pipfile environment and lock the
new dependencies. We are using avocado >= 87.0 here to take advantage of
some features that Cleber has helpfully added to make the test output
here *very* friendly and easy to read for developers that might chance
upon the output in Gitlab CI.

[Note: ALL of the dependencies get updated to the most modern versions
that exist at the time of this writing. No way around it that I have
seen. Not ideal, but so it goes.]

Provided you have the right development dependencies (mypy, flake8,
isort, pylint, and now avocado-framework) You should be able to run
"avocado --config avocado.cfg run tests/" from the python folder to run
all of these linters with the correct arguments.

(A forthcoming commit adds the much easier 'make check'.)

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/README.rst  |  2 ++
 python/Pipfile.lock|  8 
 python/avocado.cfg | 10 ++
 python/setup.cfg   |  1 +
 python/tests/flake8.sh |  2 ++
 python/tests/isort.sh  |  2 ++
 python/tests/mypy.sh   |  2 ++
 python/tests/pylint.sh |  2 ++
 8 files changed, 29 insertions(+)
 create mode 100644 python/avocado.cfg
 create mode 100755 python/tests/flake8.sh
 create mode 100755 python/tests/isort.sh
 create mode 100755 python/tests/mypy.sh
 create mode 100755 python/tests/pylint.sh

diff --git a/python/README.rst b/python/README.rst
index 954870973d0..6bd2c6b3547 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -37,6 +37,8 @@ Files in this directory
 ---
 
 - ``qemu/`` Python package source directory.
+- ``tests/`` Python package tests directory.
+- ``avocado.cfg`` Configuration for the Avocado test-runner.
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index a2cdc1c50ea..6e344f5fadf 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -30,6 +30,14 @@
 "markers": "python_version ~= '3.6'",
 "version": "==2.5.6"
 },
+"avocado-framework": {
+"hashes": [
+
"sha256:42aa7962df98d6b78d4efd9afa2177226dc630f3d83a2a7d5baf7a0a7da7fa1b",
+
"sha256:d96ae343abf890e1ef3b3a6af5ce49e35f6bded0715770c4acb325bca555c515"
+],
+"markers": "python_version >= '3.6'",
+"version": "==88.1"
+},
 "flake8": {
 "hashes": [
 
"sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b",
diff --git a/python/avocado.cfg b/python/avocado.cfg
new file mode 100644
index 000..10dc6fb6054
--- /dev/null
+++ b/python/avocado.cfg
@@ -0,0 +1,10 @@
+[simpletests]
+# Don't show stdout/stderr in the test *summary*
+status.failure_fields = ['status']
+
+[job]
+# Don't show the full debug.log output; only select stdout/stderr.
+output.testlogs.logfiles = ['stdout', 'stderr']
+
+# Show full stdout/stderr only on tests that FAIL
+output.testlogs.statuses = ['FAIL']
diff --git a/python/setup.cfg b/python/setup.cfg
index 39dc135e601..fd325194901 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -25,6 +25,7 @@ packages =
 [options.extras_require]
 # Run `pipenv lock --dev` when changing these requirements.
 devel =
+avocado-framework >= 87.0
 flake8 >= 3.6.0
 isort >= 5.1.2
 mypy >= 0.770
diff --git a/python/tests/flake8.sh b/python/tests/flake8.sh
new file mode 100755
index 000..51e0788462b
--- /dev/null
+++ b/python/tests/flake8.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m flake8
diff --git a/python/tests/isort.sh b/python/tests/isort.sh
new file mode 100755
index 000..4480405bfb0
--- /dev/null
+++ b/python/tests/isort.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m isort -c qemu/
diff --git a/python/tests/mypy.sh b/python/tests/mypy.sh
new file mode 100755
index 000..5f980f563bb
--- /dev/null
+++ b/python/tests/mypy.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m mypy -p qemu
diff --git a/python/tests/pylint.sh b/python/tests/pylint.sh
new file mode 100755
index 000..4b10b34db7c
--- /dev/null
+++ b/python/tests/pylint.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m pylint qemu/
-- 
2.31.1

[PATCH v7 22/31] python: add mypy to pipenv

[John Snow]

0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.

0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.

Now that we are checking a namespace package, we need to tell mypy to
allow PEP420 namespaces, so modify the mypy config as part of the move.

mypy can now be run from the python root by typing 'mypy -p qemu'.

A note on mypy invocation: Running it as "mypy qemu/" changes the import
path detection mechanisms in mypy slightly, and it will fail. See
https://github.com/python/mypy/issues/8584 for a decent entry point with
more breadcrumbs on the various behaviors that contribute to this subtle
difference.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/Pipfile  |  1 +
 python/Pipfile.lock | 37 -
 python/setup.cfg|  1 +
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/python/Pipfile b/python/Pipfile
index 053f344dcbe..796c6282e17 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+mypy = ">=0.770"
 pylint = ">=2.8.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 5c34019060a..626e68403f7 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
 "_meta": {
 "hash": {
-"sha256": 
"3c842ab9c72c40d24d146349aa144e00e4dec1c358c812cfa96489411f5b3f87"
+"sha256": 
"14d171b3d86759e1fdfb9e55f66be4a696b6afa8f627d6c4778f8398c6a66b98"
 },
 "pipfile-spec": 6,
 "requires": {
@@ -84,6 +84,41 @@
 ],
 "version": "==0.6.1"
 },
+"mypy": {
+"hashes": [
+
"sha256:0d0a87c0e7e3a9becdfbe936c981d32e5ee0ccda3e0f07e1ef2c3d1a817cf73e",
+
"sha256:25adde9b862f8f9aac9d2d11971f226bd4c8fbaa89fb76bdadb267ef22d10064",
+
"sha256:28fb5479c494b1bab244620685e2eb3c3f988d71fd5d64cc753195e8ed53df7c",
+
"sha256:2f9b3407c58347a452fc0736861593e105139b905cca7d097e413453a1d650b4",
+
"sha256:33f159443db0829d16f0a8d83d94df3109bb6dd801975fe86bacb9bf71628e97",
+
"sha256:3f2aca7f68580dc2508289c729bd49ee929a436208d2b2b6aab15745a70a57df",
+
"sha256:499c798053cdebcaa916eef8cd733e5584b5909f789de856b482cd7d069bdad8",
+
"sha256:4eec37370483331d13514c3f55f446fc5248d6373e7029a29ecb7b7494851e7a",
+
"sha256:552a815579aa1e995f39fd05dde6cd378e191b063f031f2acfe73ce9fb7f9e56",
+
"sha256:5873888fff1c7cf5b71efbe80e0e73153fe9212fafdf8e44adfe4c20ec9f82d7",
+
"sha256:61a3d5b97955422964be6b3baf05ff2ce7f26f52c85dd88db11d5e03e146a3a6",
+
"sha256:674e822aa665b9fd75130c6c5f5ed9564a38c6cea6a6432ce47eafb68ee578c5",
+
"sha256:7ce3175801d0ae5fdfa79b4f0cfed08807af4d075b402b7e294e6aa72af9aa2a",
+
"sha256:9743c91088d396c1a5a3c9978354b61b0382b4e3c440ce83cf77994a43e8c521",
+
"sha256:9f94aac67a2045ec719ffe6111df543bac7874cee01f41928f6969756e030564",
+
"sha256:a26f8ec704e5a7423c8824d425086705e381b4f1dfdef6e3a1edab7ba174ec49",
+
"sha256:abf7e0c3cf117c44d9285cc6128856106183938c68fd4944763003decdcfeb66",
+
"sha256:b09669bcda124e83708f34a94606e01b614fa71931d356c1f1a5297ba11f110a",
+
"sha256:cd07039aa5df222037005b08fbbfd69b3ab0b0bd7a07d7906de75ae52c4e3119",
+
"sha256:d23e0ea196702d918b60c8288561e722bf437d82cb7ef2edcd98cfa38905d506",
+
"sha256:d65cc1df038ef55a99e617431f0553cd77763869eebdf9042403e16089fe746c",
+
"sha256:d7da2e1d5f558c37d6e8c1246f1aec1e7349e4913d8fb3cb289a35de573fe2eb"
+],
+"index": "pypi",
+"version": "==0.812"
+},
+"mypy-extensions": {
+"hashes": [
+
"sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
+
"sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
+],
+"version": "==0.4.3"
+},
 "pycodestyle": {
 "hashes": [
 
"sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068",
diff --git a/python/setup.cfg b/python/setup.cfg
index bd88b44ad80..b485d6161d5 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -31,6 +31,7 @@ exclude = __pycache__,
 strict = True
 python_version = 3.6
 warn_unused_configs = True
+namespace_packages = True
 
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
-- 
2.31.1

[PATCH v7 31/31] gitlab: add python linters to CI

[John Snow]

Add a python container that contains just enough juice for us to run the python
code quality analysis tools.

Base this container on fedora, because fedora has very convenient
packaging for testing multiple python versions.

Add two tests:

check-python-pipenv uses pipenv to test a frozen, very explicit set of
packages against our minimum supported python version, Python 3.6. This
test is not allowed to fail.

check-python-tox uses tox to install the latest versions of required
python dependencies against a wide array of Python versions from 3.6 to
3.9, even including the yet-to-be-released Python 3.10. This test is
allowed to fail with a warning.

Signed-off-by: John Snow 
---
 .gitlab-ci.d/containers.yml|  5 +
 .gitlab-ci.yml | 26 ++
 tests/docker/dockerfiles/python.docker | 18 ++
 3 files changed, 49 insertions(+)
 create mode 100644 tests/docker/dockerfiles/python.docker

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 765408ae274..05ebd4dc11d 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -242,3 +242,8 @@ amd64-opensuse-leap-container:
   extends: .container_job_template
   variables:
 NAME: opensuse-leap
+
+python-container:
+  extends: .container_job_template
+  variables:
+NAME: python
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f718b61fa78..cc2a3935c62 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -789,6 +789,32 @@ check-patch:
 GIT_DEPTH: 1000
   allow_failure: true
 
+
+check-python-pipenv:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  script:
+- cd python
+- make venv-check
+  variables:
+GIT_DEPTH: 1000
+  needs:
+job: python-container
+
+
+check-python-tox:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  script:
+- cd python
+- make check-tox
+  variables:
+GIT_DEPTH: 1000
+  needs:
+job: python-container
+  allow_failure: true
+
+
 check-dco:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
diff --git a/tests/docker/dockerfiles/python.docker 
b/tests/docker/dockerfiles/python.docker
new file mode 100644
index 000..56d88417df4
--- /dev/null
+++ b/tests/docker/dockerfiles/python.docker
@@ -0,0 +1,18 @@
+# Python library testing environment
+
+FROM fedora:latest
+MAINTAINER John Snow 
+
+# Please keep this list sorted alphabetically
+ENV PACKAGES \
+gcc \
+make \
+pipenv \
+python3 \
+python3-pip \
+python3-tox \
+python3-virtualenv \
+python3.10
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt
-- 
2.31.1

[PATCH v7 25/31] python/qemu: add qemu package itself to pipenv

[John Snow]

This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)

The qemu packages are installed in 'editable' mode; all changes made to
the python package inside the git tree will be reflected in the
installed package without reinstallation. This includes changes made
via git pull and so on.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/Pipfile  | 1 +
 python/Pipfile.lock | 9 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index 79c74dd8db4..dbe96f71c48 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -10,6 +10,7 @@ mypy = ">=0.770"
 pylint = ">=2.8.0"
 
 [packages]
+qemu = {editable = true,path = "."}
 
 [requires]
 python_version = "3.6"
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 57a5befb104..f0bf576c31e 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
 "_meta": {
 "hash": {
-"sha256": 
"8173290ad57aab0b722c9b4f109519de4e0dd7cd1bad1e16806b78bb169bce08"
+"sha256": 
"7c74cc4c2db3a75c954a6686411cda6fd60e464620bb6d5f1ed9a54be61db4cc"
 },
 "pipfile-spec": 6,
 "requires": {
@@ -15,7 +15,12 @@
 }
 ]
 },
-"default": {},
+"default": {
+"qemu": {
+"editable": true,
+"path": "."
+}
+},
 "develop": {
 "astroid": {
 "hashes": [
-- 
2.31.1

[PATCH v7 24/31] python/qemu: add isort to pipenv

[John Snow]

isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.

isort < 5.1.1 has a bug where it does not handle comments near import
statements correctly.

Require 5.1.2 or greater.

isort can be run (in "check" mode) with 'isort -c qemu' from the python
root. isort can also be used to fix/rewrite import order automatically
by using 'isort qemu'.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/Pipfile  | 1 +
 python/Pipfile.lock | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index 796c6282e17..79c74dd8db4 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+isort = ">=5.1.2"
 mypy = ">=0.770"
 pylint = ">=2.8.0"
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 626e68403f7..57a5befb104 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
 "_meta": {
 "hash": {
-"sha256": 
"14d171b3d86759e1fdfb9e55f66be4a696b6afa8f627d6c4778f8398c6a66b98"
+"sha256": 
"8173290ad57aab0b722c9b4f109519de4e0dd7cd1bad1e16806b78bb169bce08"
 },
 "pipfile-spec": 6,
 "requires": {
@@ -46,7 +46,7 @@
 
"sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
 
"sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
 ],
-"markers": "python_version >= '3.6' and python_version < '4.0'",
+"index": "pypi",
 "version": "==5.8.0"
 },
 "lazy-object-proxy": {
-- 
2.31.1

[PATCH v7 19/31] python: add excluded dirs to flake8 config

[John Snow]

Instruct flake8 to avoid certain well-known directories created by
python tooling that it ought not check.

Note that at-present, nothing actually creates a ".venv" directory; but
it is in such widespread usage as a de-facto location for a developer's
virtual environment that it should be excluded anyway. A forthcoming
commit canonizes this with a "make venv" command.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/setup.cfg | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/python/setup.cfg b/python/setup.cfg
index 52a89a0a290..9aeab2bb0d3 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -24,6 +24,8 @@ packages =
 
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+exclude = __pycache__,
+  .venv,
 
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
-- 
2.31.1

[PATCH v7 23/31] python: move .isort.cfg into setup.cfg

[John Snow]

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/.isort.cfg | 7 ---
 python/setup.cfg  | 8 
 2 files changed, 8 insertions(+), 7 deletions(-)
 delete mode 100644 python/.isort.cfg

diff --git a/python/.isort.cfg b/python/.isort.cfg
deleted file mode 100644
index 6d0fd6cc0d3..000
--- a/python/.isort.cfg
+++ /dev/null
@@ -1,7 +0,0 @@
-[settings]
-force_grid_wrap=4
-force_sort_within_sections=True
-include_trailing_comma=True
-line_length=72
-lines_after_imports=2
-multi_line_output=3
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index b485d6161d5..3f07bd2752d 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -61,3 +61,11 @@ good-names=i,
 [pylint.similarities]
 # Ignore imports when computing similarities.
 ignore-imports=yes
+
+[isort]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
-- 
2.31.1

[PATCH v7 20/31] python: Add flake8 to pipenv

[John Snow]

flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.

flake8 can be run from the python/ directory with no arguments.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/Pipfile  |  1 +
 python/Pipfile.lock | 51 -
 2 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/python/Pipfile b/python/Pipfile
index 285e2c8e671..053f344dcbe 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple";
 verify_ssl = true
 
 [dev-packages]
+flake8 = ">=3.6.0"
 pylint = ">=2.8.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index c9debd09503..5c34019060a 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
 "_meta": {
 "hash": {
-"sha256": 
"bd4fb76fcdd145bbf23c3a9dd7ad966113c5ce43ca51cc2d828aa7e73d572901"
+"sha256": 
"3c842ab9c72c40d24d146349aa144e00e4dec1c358c812cfa96489411f5b3f87"
 },
 "pipfile-spec": 6,
 "requires": {
@@ -25,6 +25,22 @@
 "markers": "python_version ~= '3.6'",
 "version": "==2.5.6"
 },
+"flake8": {
+"hashes": [
+
"sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b",
+
"sha256:bf8fd46d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907"
+],
+"index": "pypi",
+"version": "==3.9.2"
+},
+"importlib-metadata": {
+"hashes": [
+
"sha256:8c501196e49fb9df5df43833bdb1e4328f64847763ec8a50703148b73784d581",
+
"sha256:d7eb1dea6d6a6086f8be21784cc9e3bcfa55872b52309bc5fad53a8ea65d"
+],
+"markers": "python_version < '3.8'",
+"version": "==4.0.1"
+},
 "isort": {
 "hashes": [
 
"sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
@@ -68,6 +84,22 @@
 ],
 "version": "==0.6.1"
 },
+"pycodestyle": {
+"hashes": [
+
"sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068",
+
"sha256:c389c1d06bf7904078ca03399a4816f974a1d590090fecea0c63ec26ebaf1cef"
+],
+"markers": "python_version >= '2.7' and python_version not in 
'3.0, 3.1, 3.2, 3.3'",
+"version": "==2.7.0"
+},
+"pyflakes": {
+"hashes": [
+
"sha256:7893783d01b8a89811dd72d7dfd4d84ff098e5eed95cfa8905b22bbffe52efc3",
+
"sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db"
+],
+"markers": "python_version >= '2.7' and python_version not in 
'3.0, 3.1, 3.2, 3.3'",
+"version": "==2.3.1"
+},
 "pylint": {
 "hashes": [
 
"sha256:586d8fa9b1891f4b725f587ef267abe2a1bad89d6b184520c7f07a253dd6e217",
@@ -120,11 +152,28 @@
 "markers": "implementation_name == 'cpython' and python_version < 
'3.8'",
 "version": "==1.4.3"
 },
+"typing-extensions": {
+"hashes": [
+
"sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497",
+
"sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342",
+
"sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84"
+],
+"markers": "python_version < '3.8'",
+"version": "==3.10.0.0"
+},
 "wrapt": {
 "hashes": [
 
"sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
 ],
 "version": "==1.12.1"
+},
+"zipp": {
+"hashes": [
+
"sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76",
+
"sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"
+],
+"markers": "python_version >= '3.6'",
+"version": "==3.4.1"
 }
 }
 }
-- 
2.31.1

[PATCH v7 18/31] python: move flake8 config to setup.cfg

[John Snow]

Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/qemu/machine/.flake8 | 2 --
 python/setup.cfg| 3 +++
 2 files changed, 3 insertions(+), 2 deletions(-)
 delete mode 100644 python/qemu/machine/.flake8

diff --git a/python/qemu/machine/.flake8 b/python/qemu/machine/.flake8
deleted file mode 100644
index 45d8146f3f5..000
--- a/python/qemu/machine/.flake8
+++ /dev/null
@@ -1,2 +0,0 @@
-[flake8]
-extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index 36b4253e939..52a89a0a290 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -22,6 +22,9 @@ packages =
 qemu.machine
 qemu.utils
 
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.31.1

[PATCH v7 21/31] python: move mypy.ini into setup.cfg

[John Snow]

mypy supports reading its configuration values from a central project
configuration file; do so.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/mypy.ini  | 4 
 python/setup.cfg | 5 +
 2 files changed, 5 insertions(+), 4 deletions(-)
 delete mode 100644 python/mypy.ini

diff --git a/python/mypy.ini b/python/mypy.ini
deleted file mode 100644
index 1a581c5f1ea..000
--- a/python/mypy.ini
+++ /dev/null
@@ -1,4 +0,0 @@
-[mypy]
-strict = True
-python_version = 3.6
-warn_unused_configs = True
diff --git a/python/setup.cfg b/python/setup.cfg
index 9aeab2bb0d3..bd88b44ad80 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -27,6 +27,11 @@ extend-ignore = E722  # Prefer pylint's bare-except checks 
to flake8's
 exclude = __pycache__,
   .venv,
 
+[mypy]
+strict = True
+python_version = 3.6
+warn_unused_configs = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.31.1

[PATCH v7 17/31] python: add pylint to pipenv

[John Snow]

We are specifying >= pylint 2.8.x for several reasons:

1. For setup.cfg support, added in pylint 2.5.x
2. To specify a version that has incompatibly dropped
   bad-whitespace checks (2.6.x)
3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9
4. 2.8.x adds a new, incompatible 'consider-using-with'
   warning that must be disabled in some cases.
   These pragmas cause warnings themselves in 2.7.x.

Signed-off-by: John Snow 
---
 python/Pipfile  |   1 +
 python/Pipfile.lock | 130 
 2 files changed, 131 insertions(+)
 create mode 100644 python/Pipfile.lock

diff --git a/python/Pipfile b/python/Pipfile
index 9534830b5eb..285e2c8e671 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple";
 verify_ssl = true
 
 [dev-packages]
+pylint = ">=2.8.0"
 
 [packages]
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
new file mode 100644
index 000..c9debd09503
--- /dev/null
+++ b/python/Pipfile.lock
@@ -0,0 +1,130 @@
+{
+"_meta": {
+"hash": {
+"sha256": 
"bd4fb76fcdd145bbf23c3a9dd7ad966113c5ce43ca51cc2d828aa7e73d572901"
+},
+"pipfile-spec": 6,
+"requires": {
+"python_version": "3.6"
+},
+"sources": [
+{
+"name": "pypi",
+"url": "https://pypi.org/simple";,
+"verify_ssl": true
+}
+]
+},
+"default": {},
+"develop": {
+"astroid": {
+"hashes": [
+
"sha256:4db03ab5fc3340cf619dbc25e42c2cc3755154ce6009469766d7143d1fc2ee4e",
+
"sha256:8a398dfce302c13f14bab13e2b14fe385d32b73f4e4853b9bdfb64598baa1975"
+],
+"markers": "python_version ~= '3.6'",
+"version": "==2.5.6"
+},
+"isort": {
+"hashes": [
+
"sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
+
"sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
+],
+"markers": "python_version >= '3.6' and python_version < '4.0'",
+"version": "==5.8.0"
+},
+"lazy-object-proxy": {
+"hashes": [
+
"sha256:17e0967ba374fc24141738c69736da90e94419338fd4c7c7bef01ee26b339653",
+
"sha256:1fee665d2638491f4d6e55bd483e15ef21f6c8c2095f235fef72601021e64f61",
+
"sha256:22ddd618cefe54305df49e4c069fa65715be4ad0e78e8d252a33debf00f6ede2",
+
"sha256:24a5045889cc2729033b3e604d496c2b6f588c754f7a62027ad4437a7ecc4837",
+
"sha256:410283732af311b51b837894fa2f24f2c0039aa7f220135192b38fcc42bd43d3",
+
"sha256:4732c765372bd78a2d6b2150a6e99d00a78ec963375f236979c0626b97ed8e43",
+
"sha256:489000d368377571c6f982fba6497f2aa13c6d1facc40660963da62f5c379726",
+
"sha256:4f60460e9f1eb632584c9685bccea152f4ac2130e299784dbaf9fae9f49891b3",
+
"sha256:5743a5ab42ae40caa8421b320ebf3a998f89c85cdc8376d6b2e00bd12bd1b587",
+
"sha256:85fb7608121fd5621cc4377a8961d0b32ccf84a7285b4f1d21988b2eae2868e8",
+
"sha256:9698110e36e2df951c7c36b6729e96429c9c32b3331989ef19976592c5f3c77a",
+
"sha256:9d397bf41caad3f489e10774667310d73cb9c4258e9aed94b9ec734b34b495fd",
+
"sha256:b579f8acbf2bdd9ea200b1d5dea36abd93cabf56cf626ab9c744a432e15c815f",
+
"sha256:b865b01a2e7f96db0c5d12cfea590f98d8c5ba64ad222300d93ce6ff9138bcad",
+
"sha256:bf34e368e8dd976423396555078def5cfc3039ebc6fc06d1ae2c5a65eebbcde4",
+
"sha256:c6938967f8528b3668622a9ed3b31d145fab161a32f5891ea7b84f6b790be05b",
+
"sha256:d1c2676e3d840852a2de7c7d5d76407c772927addff8d742b9808fe0afccebdf",
+
"sha256:d7124f52f3bd259f510651450e18e0fd081ed82f3c08541dffc7b94b883aa981",
+
"sha256:d900d949b707778696fdf01036f58c9876a0d8bfe116e8d220cfd4b15f14e741",
+
"sha256:ebfd274dcd5133e0afae738e6d9da4323c3eb021b3e13052d8cbd0e457b1256e",
+
"sha256:ed361bb83436f117f9917d282a456f9e5009ea12fd6de8742d1a4752c3017e93",
+
"sha256:f5144c75445ae3ca2057faac03fda5a902eff196702b0a24daf1d6ce0650514b"
+],
+"markers": "python_version >= '2.7' and python_version not in 
'3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
+"version": "==1.6.0"
+},
+"mccabe": {
+"hashes": [
+
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
+
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
+],
+"version": "==0.6.1"
+},
+"pylint": {
+"hashes": [
+
"sha256:586d8fa9b1891f4b725f587ef267abe2a1bad89d6b184520c7f07a253dd6e217",
+
"sha256:f7e2072654a6b6afdf5e2fb38147d3e2d2d43c89f648637baab63e026

[PATCH v7 15/31] python: add pylint import exceptions

[John Snow]

Pylint 2.5.x - 2.7.x have regressions that make import checking
inconsistent, see:

https://github.com/PyCQA/pylint/issues/3609
https://github.com/PyCQA/pylint/issues/3624
https://github.com/PyCQA/pylint/issues/3651

Pinning to 2.4.4 is worse, because it mandates versions of shared
dependencies that are too old for features we want in isort and mypy.
Oh well.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/qemu/machine/__init__.py | 3 +++
 python/qemu/machine/machine.py  | 2 +-
 python/qemu/machine/qtest.py| 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/python/qemu/machine/__init__.py b/python/qemu/machine/__init__.py
index 98302ea31e7..728f27adbed 100644
--- a/python/qemu/machine/__init__.py
+++ b/python/qemu/machine/__init__.py
@@ -22,6 +22,9 @@
 # the COPYING file in the top-level directory.
 #
 
+# pylint: disable=import-error
+# see: https://github.com/PyCQA/pylint/issues/3624
+# see: https://github.com/PyCQA/pylint/issues/3651
 from .machine import QEMUMachine
 from .qtest import QEMUQtestMachine, QEMUQtestProtocol
 
diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index d33b02d2ce6..b62435528e2 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -38,7 +38,7 @@
 Type,
 )
 
-from qemu.qmp import (
+from qemu.qmp import (  # pylint: disable=import-error
 QEMUMonitorProtocol,
 QMPMessage,
 QMPReturnValue,
diff --git a/python/qemu/machine/qtest.py b/python/qemu/machine/qtest.py
index e893ca3697a..93700684d1c 100644
--- a/python/qemu/machine/qtest.py
+++ b/python/qemu/machine/qtest.py
@@ -26,7 +26,7 @@
 TextIO,
 )
 
-from qemu.qmp import SocketAddrT
+from qemu.qmp import SocketAddrT  # pylint: disable=import-error
 
 from .machine import QEMUMachine
 
-- 
2.31.1

[PATCH v7 13/31] python: add MANIFEST.in

[John Snow]

When creating a source or binary distribution via 'python3 setup.py
', the VERSION and PACKAGE.rst files aren't bundled by
default. Create a MANIFEST.in file that instructs the build tools to
include these so that installation from these files won't fail.

This is required by 'tox', as well as by the tooling needed to upload
packages to PyPI.

Exclude the 'README.rst' file -- that's intended as a guidebook to our
source tree, not a file that needs to be distributed.

Signed-off-by: John Snow 
---
 python/README.rst  | 2 ++
 python/MANIFEST.in | 3 +++
 2 files changed, 5 insertions(+)
 create mode 100644 python/MANIFEST.in

diff --git a/python/README.rst b/python/README.rst
index 38b0c83f321..0099646ae2f 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -33,6 +33,8 @@ Files in this directory
 ---
 
 - ``qemu/`` Python package source directory.
+- ``MANIFEST.in`` is read by python setuptools, it specifies additional files
+  that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
new file mode 100644
index 000..7059ad28221
--- /dev/null
+++ b/python/MANIFEST.in
@@ -0,0 +1,3 @@
+include VERSION
+include PACKAGE.rst
+exclude README.rst
-- 
2.31.1

[PATCH v7 16/31] python: move pylintrc into setup.cfg

[John Snow]

Delete the empty settings now that it's sharing a home with settings for
other tools.

pylint can now be run from this folder as "pylint qemu".

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 python/qemu/machine/pylintrc | 58 
 python/setup.cfg | 29 ++
 2 files changed, 29 insertions(+), 58 deletions(-)
 delete mode 100644 python/qemu/machine/pylintrc

diff --git a/python/qemu/machine/pylintrc b/python/qemu/machine/pylintrc
deleted file mode 100644
index 3f69205000d..000
--- a/python/qemu/machine/pylintrc
+++ /dev/null
@@ -1,58 +0,0 @@
-[MASTER]
-
-[MESSAGES CONTROL]
-
-# Disable the message, report, category or checker with the given id(s). You
-# can either give multiple identifiers separated by comma (,) or put this
-# option multiple times (only on the command line, not in the configuration
-# file where it should appear only once). You can also use "--disable=all" to
-# disable everything first and then reenable specific checks. For example, if
-# you want to run only the similarities checker, you can use "--disable=all
-# --enable=similarities". If you want to run only the classes checker, but have
-# no Warning level messages displayed, use "--disable=all --enable=classes
-# --disable=W".
-disable=too-many-arguments,
-too-many-instance-attributes,
-too-many-public-methods,
-
-[REPORTS]
-
-[REFACTORING]
-
-[MISCELLANEOUS]
-
-[LOGGING]
-
-[BASIC]
-
-# Good variable names which should always be accepted, separated by a comma.
-good-names=i,
-   j,
-   k,
-   ex,
-   Run,
-   _,
-   fd,
-   c,
-[VARIABLES]
-
-[STRING]
-
-[SPELLING]
-
-[FORMAT]
-
-[SIMILARITIES]
-
-# Ignore imports when computing similarities.
-ignore-imports=yes
-
-[TYPECHECK]
-
-[CLASSES]
-
-[IMPORTS]
-
-[DESIGN]
-
-[EXCEPTIONS]
diff --git a/python/setup.cfg b/python/setup.cfg
index b0010e0188f..36b4253e939 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -21,3 +21,32 @@ packages =
 qemu.qmp
 qemu.machine
 qemu.utils
+
+[pylint.messages control]
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=too-many-arguments,
+too-many-instance-attributes,
+too-many-public-methods,
+
+[pylint.basic]
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+   j,
+   k,
+   ex,
+   Run,
+   _,
+   fd,
+   c,
+
+[pylint.similarities]
+# Ignore imports when computing similarities.
+ignore-imports=yes
-- 
2.31.1

[PATCH v7 10/31] python: add qemu package installer

[John Snow]

Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
this package is for and who to contact for more information. This
document will be used as the landing page for the package on PyPI.

List the subpackages we intend to package by name instead of using
find_namespace because find_namespace will naively also packages tests,
things it finds in the dist/ folder, etc. I could not figure out how to
modify this behavior; adding allow/deny lists to setuptools kept
changing the packaged hierarchy. This works, roll with it.

I am not yet using a pyproject.toml style package manifest, because
"editable" installs are not defined (yet?) by PEP-517/518.

I consider editable installs crucial for development, though they have
(apparently) always been somewhat poorly defined.

Pip now (19.2 and later) now supports editable installs for projects
using pyproject.toml manifests, but might require the use of the
--no-use-pep517 flag, which somewhat defeats the point. Full support for
setup.py-less editable installs was not introduced until pip 21.1.1:
https://github.com/pypa/pip/pull/9547/commits/7a95720e796a5e56481c1cc20b6ce6249c50f357

For now, while the dust settles, stick with the de-facto
setup.py/setup.cfg combination supported by setuptools. It will be worth
re-evaluating this point again in the future when our supported build
platforms all ship a fairly modern pip.

Additional reading on this matter:

https://github.com/pypa/packaging-problems/issues/256
https://github.com/pypa/pip/issues/6334
https://github.com/pypa/pip/issues/6375
https://github.com/pypa/pip/issues/6434
https://github.com/pypa/pip/issues/6438

Signed-off-by: John Snow 
---
 python/PACKAGE.rst | 33 +
 python/setup.cfg   | 22 ++
 python/setup.py| 23 +++
 3 files changed, 78 insertions(+)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/setup.cfg
 create mode 100755 python/setup.py

diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
new file mode 100644
index 000..1bbfe1b58e2
--- /dev/null
+++ b/python/PACKAGE.rst
@@ -0,0 +1,33 @@
+QEMU Python Tooling
+===
+
+This package provides QEMU tooling used by the QEMU project to build,
+configure, and test QEMU. It is not a fully-fledged SDK and it is subject
+to change at any time.
+
+Usage
+-
+
+The ``qemu.qmp`` subpackage provides a library for communicating with
+QMP servers. The ``qemu.machine`` subpackage offers rudimentary
+facilities for launching and managing QEMU processes. Refer to each
+package's documentation
+(``>>> help(qemu.qmp)``, ``>>> help(qemu.machine)``)
+for more information.
+
+Contributing
+
+
+This package is maintained by John Snow  as part of
+the QEMU source tree. Contributions are welcome and follow the `QEMU
+patch submission process
+`_, which involves
+sending patches to the QEMU development mailing list.
+
+John maintains a `GitLab staging branch
+`_, and there is an
+official `GitLab mirror `_.
+
+Please report bugs on the `QEMU issue tracker
+`_ and tag ``@jsnow`` in
+the report.
diff --git a/python/setup.cfg b/python/setup.cfg
new file mode 100644
index 000..3fa92a2e73f
--- /dev/null
+++ b/python/setup.cfg
@@ -0,0 +1,22 @@
+[metadata]
+name = qemu
+maintainer = QEMU Developer Team
+maintainer_email = qemu-devel@nongnu.org
+url = https://www.qemu.org/
+download_url = https://www.qemu.org/download/
+description = QEMU Python Build, Debug and SDK tooling.
+long_description = file:PACKAGE.rst
+long_description_content_type = text/x-rst
+classifiers =
+Development Status :: 3 - Alpha
+License :: OSI Approved :: GNU General Public License v2 (GPLv2)
+Natural Language :: English
+Operating System :: OS Independent
+Programming Language :: Python :: 3 :: Only
+
+[options]
+python_requires = >= 3.6
+packages =
+qemu.qmp
+qemu.machine
+qemu.utils
diff --git a/python/setup.py b/python/setup.py
new file mode 100755
index 000..2014f81b757
--- /dev/null
+++ b/python/setup.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+"""
+QEMU tooling installer script
+Copyright (c) 2020-2021 John Snow for Red Hat, Inc.
+"""
+
+import setuptools
+import pkg_resources
+
+
+def main():
+"""
+QEMU tooling installer
+"""
+
+# 
https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
+pkg_resources.require('setuptools>=39.2')
+
+setuptools.setup()
+
+
+if __name__ == '__main__':
+main()
-- 
2.31.1

[PATCH v7 06/31] python/machine: disable warning for Popen in _launch()

[John Snow]

We handle this resource rather meticulously in
shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in
_do_shutdown().

Quiet this pylint warning here.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Message-id: 20210517184808.3562549-7-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/machine.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index c66bc6a9c69..5d72c4ca369 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -405,6 +405,9 @@ def _launch(self) -> None:
   self._args)
 )
 LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
+
+# Cleaning up of this subprocess is guaranteed by _do_shutdown.
+# pylint: disable=consider-using-with
 self._popen = subprocess.Popen(self._qemu_full_args,
stdin=subprocess.DEVNULL,
stdout=self._qemu_log_file,
-- 
2.31.1

[PATCH v7 14/31] python: Add pipenv support

[John Snow]

pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.

pipenv uses two files to do this:

(1) Pipfile, which is similar in purpose and scope to what setup.cfg
lists. It specifies the requisite minimum to get a functional
environment for using this package.

(2) Pipfile.lock, which is similar in purpose to `pip freeze >
requirements.txt`. It specifies a canonical virtual environment used for
deployment or testing. This ensures that all users have repeatable
results.

The primary benefit of using this tool is to ensure *rock solid*
repeatable CI results with a known set of packages. Although I endeavor
to support as many versions as I can, the fluid nature of the Python
toolchain often means tailoring code for fairly specific versions.

Note that pipenv is *not* required to install or use this module; this is
purely for the sake of repeatable testing by CI or developers.

Here, a "blank" pipfile is added with no dependencies, but specifies
Python 3.6 for the virtual environment.

Pipfile will specify our version minimums, while Pipfile.lock specifies
an exact loadout of packages that were known to operate correctly. This
latter file provides the real value for easy setup of container images
and CI environments.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/README.rst |  3 +++
 python/Pipfile| 11 +++
 2 files changed, 14 insertions(+)
 create mode 100644 python/Pipfile

diff --git a/python/README.rst b/python/README.rst
index 0099646ae2f..bf9bbca979a 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -36,6 +36,9 @@ Files in this directory
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
+- ``Pipfile`` is used by Pipenv to generate ``Pipfile.lock``.
+- ``Pipfile.lock`` is a set of pinned package dependencies that this package
+  is tested under in our CI suite. It is used by ``make venv-check``.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
   this package; it is referenced by ``setup.cfg``.
diff --git a/python/Pipfile b/python/Pipfile
new file mode 100644
index 000..9534830b5eb
--- /dev/null
+++ b/python/Pipfile
@@ -0,0 +1,11 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple";
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+
+[requires]
+python_version = "3.6"
-- 
2.31.1

[PATCH v7 05/31] python/machine: Disable pylint warning for open() in _pre_launch

[John Snow]

Shift the open() call later so that the pylint pragma applies *only* to
that one open() call. Add a note that suggests why this is safe: the
resource is unconditionally cleaned up in _post_shutdown().

_post_shutdown is called after failed launches (see launch()), and
unconditionally after every call to shutdown(), and therefore also on
__exit__.

Signed-off-by: John Snow 
Reviewed-by: Wainer dos Santos Moschetta 
Reviewed-by: Cleber Rosa 
Message-id: 20210517184808.3562549-6-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/machine.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index 04e005f3811..c66bc6a9c69 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -306,7 +306,6 @@ def _base_args(self) -> List[str]:
 
 def _pre_launch(self) -> None:
 self._qemu_log_path = os.path.join(self.temp_dir, self._name + ".log")
-self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
 if self._console_set:
 self._remove_files.append(self._console_address)
@@ -321,6 +320,11 @@ def _pre_launch(self) -> None:
 nickname=self._name
 )
 
+# NOTE: Make sure any opened resources are *definitely* freed in
+# _post_shutdown()!
+# pylint: disable=consider-using-with
+self._qemu_log_file = open(self._qemu_log_path, 'wb')
+
 def _post_launch(self) -> None:
 if self._qmp_connection:
 self._qmp.accept()
-- 
2.31.1

[PATCH v7 12/31] python: add directory structure README.rst files

[John Snow]

Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source tree.

They are not designed for inclusion in a published manual.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/README.rst  | 41 ++
 python/qemu/README.rst |  8 +++
 python/qemu/machine/README.rst |  9 
 python/qemu/qmp/README.rst |  9 
 python/qemu/utils/README.rst   |  7 ++
 5 files changed, 74 insertions(+)
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/qemu/utils/README.rst

diff --git a/python/README.rst b/python/README.rst
new file mode 100644
index 000..38b0c83f321
--- /dev/null
+++ b/python/README.rst
@@ -0,0 +1,41 @@
+QEMU Python Tooling
+===
+
+This directory houses Python tooling used by the QEMU project to build,
+configure, and test QEMU. It is organized by namespace (``qemu``), and
+then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
+
+``setup.py`` is used by ``pip`` to install this tooling to the current
+environment. ``setup.cfg`` provides the packaging configuration used by
+``setup.py`` in a setuptools specific format. You will generally invoke
+it by doing one of the following:
+
+1. ``pip3 install .`` will install these packages to your current
+   environment. If you are inside a virtual environment, they will
+   install there. If you are not, it will attempt to install to the
+   global environment, which is **not recommended**.
+
+2. ``pip3 install --user .`` will install these packages to your user's
+   local python packages. If you are inside of a virtual environment,
+   this will fail; you likely want the first invocation above.
+
+If you append the ``-e`` argument, pip will install in "editable" mode;
+which installs a version of the package that installs a forwarder
+pointing to these files, such that the package always reflects the
+latest version in your git tree.
+
+See `Installing packages using pip and virtual environments
+`_
+for more information.
+
+
+Files in this directory
+---
+
+- ``qemu/`` Python package source directory.
+- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
+- ``README.rst`` you are here!
+- ``VERSION`` contains the PEP-440 compliant version used to describe
+  this package; it is referenced by ``setup.cfg``.
+- ``setup.cfg`` houses setuptools package configuration.
+- ``setup.py`` is the setuptools installer used by pip; See above.
diff --git a/python/qemu/README.rst b/python/qemu/README.rst
new file mode 100644
index 000..d04943f526c
--- /dev/null
+++ b/python/qemu/README.rst
@@ -0,0 +1,8 @@
+QEMU Python Namespace
+=
+
+This directory serves as the root of a `Python PEP 420 implicit
+namespace package `_.
+
+Each directory below is assumed to be an installable Python package that
+is available under the ``qemu.`` namespace.
diff --git a/python/qemu/machine/README.rst b/python/qemu/machine/README.rst
new file mode 100644
index 000..ac2b4fffb42
--- /dev/null
+++ b/python/qemu/machine/README.rst
@@ -0,0 +1,9 @@
+qemu.machine package
+
+
+This package provides core utilities used for testing and debugging
+QEMU. It is used by the iotests, vm tests, acceptance tests, and several
+other utilities in the ./scripts directory. It is not a fully-fledged
+SDK and it is subject to change at any time.
+
+See the documentation in ``__init__.py`` for more information.
diff --git a/python/qemu/qmp/README.rst b/python/qemu/qmp/README.rst
new file mode 100644
index 000..c21951491cf
--- /dev/null
+++ b/python/qemu/qmp/README.rst
@@ -0,0 +1,9 @@
+qemu.qmp package
+
+
+This package provides a library used for connecting to and communicating
+with QMP servers. It is used extensively by iotests, vm tests,
+acceptance tests, and other utilities in the ./scripts directory. It is
+not a fully-fledged SDK and is subject to change at any time.
+
+See the documentation in ``__init__.py`` for more information.
diff --git a/python/qemu/utils/README.rst b/python/qemu/utils/README.rst
new file mode 100644
index 000..975fbf4d7de
--- /dev/null
+++ b/python/qemu/utils/README.rst
@@ -0,0 +1,7 @@
+qemu.utils package
+==
+
+This package provides miscellaneous utilities used for testing and
+debugging QEMU. It is used primarily by the vm and acceptance tests.
+
+See the documentation in ``__init__.py`` for more information.
-- 
2.31.1

[PATCH v7 02/31] python/machine: use subprocess.DEVNULL instead of open(os.path.devnull)

[John Snow]

One less file resource to manage, and it helps quiet some pylint >=
2.8.0 warnings about not using a with-context manager for the open call.

Signed-off-by: John Snow 
Reviewed-by: Philippe Mathieu-Daudé 
Reviewed-by: Cleber Rosa 
Message-id: 20210517184808.3562549-3-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/machine.py | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index b379fcbe726..5b87e9ce024 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -223,9 +223,8 @@ def send_fd_scm(self, fd: Optional[int] = None,
 assert fd is not None
 fd_param.append(str(fd))
 
-devnull = open(os.path.devnull, 'rb')
 proc = subprocess.Popen(
-fd_param, stdin=devnull, stdout=subprocess.PIPE,
+fd_param, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
 stderr=subprocess.STDOUT, close_fds=False
 )
 output = proc.communicate()[0]
@@ -391,7 +390,6 @@ def _launch(self) -> None:
 """
 Launch the VM and establish a QMP connection
 """
-devnull = open(os.path.devnull, 'rb')
 self._pre_launch()
 self._qemu_full_args = tuple(
 chain(self._wrapper,
@@ -401,7 +399,7 @@ def _launch(self) -> None:
 )
 LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
 self._popen = subprocess.Popen(self._qemu_full_args,
-   stdin=devnull,
+   stdin=subprocess.DEVNULL,
stdout=self._qemu_log_file,
stderr=subprocess.STDOUT,
shell=False,
-- 
2.31.1

[PATCH v7 11/31] python: add VERSION file

[John Snow]

Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible.

The problem is that when using pip to build and install a python
package, it copies files over to a temporary directory and performs its
build there. This loses access to any information in the parent
directory, including git itself.

Further, Python versions have a standard (PEP 440) that may or may not
follow QEMU's versioning. In general, it does; but naturally QEMU does
not follow PEP 440. To avoid any automatically-generated conflict, a
manual version file is preferred.


I am proposing:

- Python tooling follows the QEMU version, indirectly, but with a major
  version of 0 to indicate that the API is not expected to be
  stable. This would mean version 0.5.2.0, 0.5.1.1, 0.5.3.0, etc.

- In the event that a Python package needs to be updated independently
  of the QEMU version, a pre-release alpha version should be preferred,
  but *only* after inclusion to the qemu development or stable branches.

  e.g. 0.5.2.0a1, 0.5.2.0a2, and so on should be preferred prior to
  5.2.0's release.

- The Python core tooling makes absolutely no version compatibility
  checks or constraints. It *may* work with releases of QEMU from the
  past or future, but it is not required to.

  i.e., "qemu.machine" will, for now, remain in lock-step with QEMU.

- We reserve the right to split the qemu package into independently
  versioned subpackages at a later date. This might allow for us to
  begin versioning QMP independently from QEMU at a later date, if
  we so choose.


Implement this versioning scheme by adding a VERSION file and setting it
to 0.6.0.0a1.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 python/VERSION   | 1 +
 python/setup.cfg | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 python/VERSION

diff --git a/python/VERSION b/python/VERSION
new file mode 100644
index 000..c19f3b832b7
--- /dev/null
+++ b/python/VERSION
@@ -0,0 +1 @@
+0.6.1.0a1
diff --git a/python/setup.cfg b/python/setup.cfg
index 3fa92a2e73f..b0010e0188f 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -1,5 +1,6 @@
 [metadata]
 name = qemu
+version = file:VERSION
 maintainer = QEMU Developer Team
 maintainer_email = qemu-devel@nongnu.org
 url = https://www.qemu.org/
-- 
2.31.1

[PATCH v7 04/31] python/console_socket: Add a pylint ignore

[John Snow]

We manage cleaning up this resource ourselves. Pylint should shush.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Message-id: 20210517184808.3562549-5-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/console_socket.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py
index 87237bebef7..8c4ff598ad7 100644
--- a/python/qemu/console_socket.py
+++ b/python/qemu/console_socket.py
@@ -39,6 +39,7 @@ def __init__(self, address: str, file: Optional[str] = None,
 self.connect(address)
 self._logfile = None
 if file:
+# pylint: disable=consider-using-with
 self._logfile = open(file, "bw")
 self._open = True
 self._drain_thread = None
-- 
2.31.1

[PATCH v7 09/31] python: create qemu packages

[John Snow]

move python/qemu/*.py to python/qemu/[machine, qmp, utils]/*.py and
update import directives across the tree.

This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into new 'machine',
'qmp' and 'utils' subpackages instead.

Implement machine/__init__.py making the top-level classes and functions
from its various modules available directly inside the package. Change
qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
library classes are available directly from "qemu.qmp" instead of
"qemu.qmp.qmp".

Signed-off-by: John Snow 
---
 python/{qemu => }/.isort.cfg|  0
 python/qemu/__init__.py | 11 ---
 python/qemu/{ => machine}/.flake8   |  0
 python/qemu/machine/__init__.py | 33 +
 python/qemu/{ => machine}/console_socket.py |  0
 python/qemu/{ => machine}/machine.py| 16 ++
 python/qemu/{ => machine}/pylintrc  |  0
 python/qemu/{ => machine}/qtest.py  |  3 +-
 python/qemu/{qmp.py => qmp/__init__.py} | 12 +++-
 python/qemu/{utils.py => utils/__init__.py} | 18 +--
 python/qemu/{ => utils}/accel.py|  0
 tests/acceptance/avocado_qemu/__init__.py   |  9 +++---
 tests/acceptance/virtio-gpu.py  |  2 +-
 tests/qemu-iotests/300  |  4 +--
 tests/qemu-iotests/iotests.py   |  2 +-
 tests/vm/aarch64vm.py   |  2 +-
 tests/vm/basevm.py  |  3 +-
 17 files changed, 83 insertions(+), 32 deletions(-)
 rename python/{qemu => }/.isort.cfg (100%)
 delete mode 100644 python/qemu/__init__.py
 rename python/qemu/{ => machine}/.flake8 (100%)
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/console_socket.py (100%)
 rename python/qemu/{ => machine}/machine.py (98%)
 rename python/qemu/{ => machine}/pylintrc (100%)
 rename python/qemu/{ => machine}/qtest.py (99%)
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
 rename python/qemu/{utils.py => utils/__init__.py} (66%)
 rename python/qemu/{ => utils}/accel.py (100%)

diff --git a/python/qemu/.isort.cfg b/python/.isort.cfg
similarity index 100%
rename from python/qemu/.isort.cfg
rename to python/.isort.cfg
diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
deleted file mode 100644
index 4ca06c34a41..000
--- a/python/qemu/__init__.py
+++ /dev/null
@@ -1,11 +0,0 @@
-# QEMU library
-#
-# Copyright (C) 2015-2016 Red Hat Inc.
-# Copyright (C) 2012 IBM Corp.
-#
-# Authors:
-#  Fam Zheng 
-#
-# This work is licensed under the terms of the GNU GPL, version 2.  See
-# the COPYING file in the top-level directory.
-#
diff --git a/python/qemu/.flake8 b/python/qemu/machine/.flake8
similarity index 100%
rename from python/qemu/.flake8
rename to python/qemu/machine/.flake8
diff --git a/python/qemu/machine/__init__.py b/python/qemu/machine/__init__.py
new file mode 100644
index 000..98302ea31e7
--- /dev/null
+++ b/python/qemu/machine/__init__.py
@@ -0,0 +1,33 @@
+"""
+QEMU development and testing library.
+
+This library provides a few high-level classes for driving QEMU from a
+test suite, not intended for production use.
+
+- QEMUMachine: Configure and Boot a QEMU VM
+ - QEMUQtestMachine: VM class, with a qtest socket.
+
+- QEMUQtestProtocol: Connect to, send/receive qtest messages.
+"""
+
+# Copyright (C) 2020-2021 John Snow for Red Hat Inc.
+# Copyright (C) 2015-2016 Red Hat Inc.
+# Copyright (C) 2012 IBM Corp.
+#
+# Authors:
+#  John Snow 
+#  Fam Zheng 
+#
+# This work is licensed under the terms of the GNU GPL, version 2.  See
+# the COPYING file in the top-level directory.
+#
+
+from .machine import QEMUMachine
+from .qtest import QEMUQtestMachine, QEMUQtestProtocol
+
+
+__all__ = (
+'QEMUMachine',
+'QEMUQtestProtocol',
+'QEMUQtestMachine',
+)
diff --git a/python/qemu/console_socket.py 
b/python/qemu/machine/console_socket.py
similarity index 100%
rename from python/qemu/console_socket.py
rename to python/qemu/machine/console_socket.py
diff --git a/python/qemu/machine.py b/python/qemu/machine/machine.py
similarity index 98%
rename from python/qemu/machine.py
rename to python/qemu/machine/machine.py
index a8837b36e47..d33b02d2ce6 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine/machine.py
@@ -38,8 +38,14 @@
 Type,
 )
 
-from . import console_socket, qmp
-from .qmp import QMPMessage, QMPReturnValue, SocketAddrT
+from qemu.qmp import (
+QEMUMonitorProtocol,
+QMPMessage,
+QMPReturnValue,
+SocketAddrT,
+)
+
+from . import console_socket
 
 
 LOG = logging.getLogger(__name__)
@@ -139,7 +145,7 @@ def __init__(self,
 self._events: List[QMPMessage] = []
 self._iolog: Optional[str] = None
 self._qmp_set = True   # Enable QMP monitor by default.
-self._qmp_connection: Optional[qmp.QEMUMonitorPro

[PATCH v7 01/31] python/console_socket: avoid one-letter variable

[John Snow]

Fixes pylint warnings.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Reviewed-by: Philippe Mathieu-Daudé 
Message-id: 20210517184808.3562549-2-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/console_socket.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py
index ac21130e446..87237bebef7 100644
--- a/python/qemu/console_socket.py
+++ b/python/qemu/console_socket.py
@@ -46,11 +46,11 @@ def __init__(self, address: str, file: Optional[str] = None,
 self._drain_thread = self._thread_start()
 
 def __repr__(self) -> str:
-s = super().__repr__()
-s = s.rstrip(">")
-s = "%s,  logfile=%s, drain_thread=%s>" % (s, self._logfile,
-   self._drain_thread)
-return s
+tmp = super().__repr__()
+tmp = tmp.rstrip(">")
+tmp = "%s,  logfile=%s, drain_thread=%s>" % (tmp, self._logfile,
+ self._drain_thread)
+return tmp
 
 def _drain_fn(self) -> None:
 """Drains the socket and runs while the socket is open."""
-- 
2.31.1

[PATCH v7 07/31] python/machine: Trim line length to below 80 chars

[John Snow]

One more little delinting fix that snuck in.

Signed-off-by: John Snow 
---
 python/qemu/machine.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index 5d72c4ca369..a8837b36e47 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -97,7 +97,7 @@ def __init__(self,
 @param args: list of extra arguments
 @param wrapper: list of arguments used as prefix to qemu binary
 @param name: prefix for socket and log file names (default: qemu-PID)
-@param base_temp_dir: default location where temporary files are 
created
+@param base_temp_dir: default location where temp files are created
 @param monitor_address: address for QMP monitor
 @param socket_scm_helper: helper program, required for send_fd_scm()
 @param sock_dir: where to create socket (defaults to base_temp_dir)
-- 
2.31.1

[PATCH v7 03/31] python/machine: use subprocess.run instead of subprocess.Popen

[John Snow]

use run() instead of Popen() -- to assert to pylint that we are not
forgetting to close a long-running program.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-id: 20210517184808.3562549-4-js...@redhat.com
Signed-off-by: John Snow 
---
 python/qemu/machine.py | 15 +--
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index 5b87e9ce024..04e005f3811 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -223,13 +223,16 @@ def send_fd_scm(self, fd: Optional[int] = None,
 assert fd is not None
 fd_param.append(str(fd))
 
-proc = subprocess.Popen(
-fd_param, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
-stderr=subprocess.STDOUT, close_fds=False
+proc = subprocess.run(
+fd_param,
+stdin=subprocess.DEVNULL,
+stdout=subprocess.PIPE,
+stderr=subprocess.STDOUT,
+check=False,
+close_fds=False,
 )
-output = proc.communicate()[0]
-if output:
-LOG.debug(output)
+if proc.stdout:
+LOG.debug(proc.stdout)
 
 return proc.returncode
 
-- 
2.31.1

[PATCH v7 08/31] iotests/297: add --namespace-packages to mypy arguments

[John Snow]

mypy is kind of weird about how it handles imports. For legacy reasons,
it won't load PEP 420 namespaces, because of logic implemented prior to
that becoming a standard.

So, if you plan on using any, you have to pass
--namespace-packages. Alright, fine.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 tests/qemu-iotests/297 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index a37910b42d9..433b7323368 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -95,6 +95,7 @@ def run_linters():
 '--warn-redundant-casts',
 '--warn-unused-ignores',
 '--no-implicit-reexport',
+'--namespace-packages',
 filename),
env=env,
check=False,
-- 
2.31.1

[PATCH v7 00/31] python: create installable package

[John Snow]

Based-on: https://gitlab.com/cleber.gnu/qemu/-/commits/python-next
CI: https://gitlab.com/jsnow/qemu/-/pipelines/309506648
GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk4
MR: https://gitlab.com/jsnow/qemu/-/merge_requests/7

ABOUT
=

This series factors the python/qemu directory as an installable
package. It does not yet actually change the mechanics of how any other
python source in the tree actually consumes it (yet), beyond the import
path -- some import statements change in a few places.

RATIONALE
=

The primary motivation of this series is primarily to formalize our
dependencies on mypy, flake8, isort, and pylint alongside versions that
are known to work. It does this using the setup.cfg and setup.py
files. It also adds explicitly pinned versions (using Pipfile.lock) of
these dependencies that should behave in a repeatable and known way for
developers and CI environments both. Lastly, it enables those CI checks
such that we can enforce Python coding quality checks via the CI tests.

An auxiliary motivation is that this package is formatted in such a way
that it COULD be uploaded to https://pypi.org/project/qemu and installed
independently of qemu.git with `pip install qemu`, but that button
remains *unpushed* and this series *will not* cause any such
releases. We have time to debate finer points like API guarantees and
versioning even after this series is merged.

Other bits of interest
--

With the python tooling as a proper package, you can install this
package in editable or production mode to a virtual environment, your
local user environment, or your system packages. The primary benefit of
this is to gain access to QMP tooling regardless of CWD, without needing
to battle sys.path (and confounding other python analysis tools).

For example: when developing, you may go to qemu/python/ and run `make
venv` followed by `pipenv shell` to activate a virtual environment that
contains the qemu python packages. These packages will always reflect
the current version of the source files in the tree. When you are
finished, you can simply exit the shell (^d) to remove these packages
from your python environment.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing by using pip: "pip install
[--user] ."

TESTING THIS SERIES
===

First of all, nothing should change. Without any intervention,
everything should behave exactly as it did before. The only new
information here comes from how to interact with and run the linters
that will be enforcing code quality standards in this subdirectory.

There are various invocations available that will test subtly different
combinations using subtly different environments. I am assuming some
light knowledge of Python environments and installing Python packages
here. If you have questions, I would be delighted to answer them.

To test the new tests, CD to ./python/ first, and then:

0. Try "make" or "make help" to get a sense of this series.

1. Try "make venv && pipenv shell" to get a venv with the package
   installed to it in editable mode. Ctrl+d exits this venv shell. While
   in this shell, any python script that uses "from qemu.[qmp|machine]
   import ..." should work correctly regardless of where the script is,
   or what your CWD is.

   This will pull some packages from PyPI and install them into the
   virtual environment, leaving your normal environment untouched.

   You will need Python 3.6 and pipenv installed on your system to do
   this step. For Fedora: "dnf install python36 pipenv" will do the
   trick. If you don't have this, skip down to \#4 and onwards.

2. Try "make check" while still in the shell to run the Python linters
using the venv built in the previous step. This will run avocado, which
will in turn execute mypy, flake8, isort and pylint with the correct
arguments.

3. Having exited the shell from above, try "make venv-check". This will
create and update the venv if needed, then run 'make check' within the
context of that shell. It should pass as long as the above did. You
should be able to run "make distclean" prior to running "make
venv-check" and have the entire process work start to finish.

4. Still outside of the venv, you may try running "make check". This
will not install anything, but unless you have the right Python
dependencies installed, these tests may fail for you. You might try
using "pip install --user .[devel]" to install the development packages
needed to run the tests successfully to your local user's python
environment. Once done, you will probably want to "pip uninstall qemu"
to remove the qemu packages to avoid them interfering with other things.

5. "make distclean" will delete the venv and any temporary files that
may have been created by packaging, installing, testing, etc.

6. You may also (if you wish) create your own environment

Re: [PATCH] tcg/aarch64/tcg-target.c.inc: correction of rotate bit number

[Richard Henderson]

On 5/25/21 2:46 AM, Yasuo Kuwahara wrote:

The last argument of tcg_out_extr() must be in the range 0-31 if ext==0.

Before the fix, when m==0 it becomes 32 and it crashes with an Illegal 
instruction in Apple Silicon.


After the fix, it will be 0. If m is in the range 1-31, it is the same as 
before.


Signed-off-by: Yasuo Kuwahara mailto:kwh...@gmail.com>>

---

tcg/aarch64/tcg-target.c.inc | 5 ++---

1 file changed, 2 insertions(+), 3 deletions(-)


(1) All patches go to qemu-devel, as per
https://wiki.qemu.org/Contribute/SubmitAPatch
Many maintainers including myself don't monitor qemu-trivial.

(2) Something is wrong with your mailer and it is adding extra
newlines, which means the patch does not apply.  However,
because the patch is so simple, I have applied it by hand.

Queued to tcg-next, thanks.


r~




diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc

index f07ba98aa4..5bd366f2d4 100644

--- a/tcg/aarch64/tcg-target.c.inc

+++ b/tcg/aarch64/tcg-target.c.inc

@@ -1291,9 +1291,8 @@ static inline void tcg_out_rotr(TCGContext *s, TCGType 
ext,

static inline void tcg_out_rotl(TCGContext *s, TCGType ext,

TCGReg rd, TCGReg rn, unsigned int m)

{

-int bits = ext ? 64 : 32;

-int max = bits - 1;

-tcg_out_extr(s, ext, rd, rn, rn, bits - (m & max));

+int max = ext ? 63 : 31;

+tcg_out_extr(s, ext, rd, rn, rn, -m & max);

}

static inline void tcg_out_dep(TCGContext *s, TCGType ext, TCGReg rd,

--

2.24.3

Re: [PATCH v6 23/25] python: add .gitignore

[John Snow]

On 5/25/21 4:42 PM, Cleber Rosa wrote:

On Tue, May 25, 2021 at 04:10:55PM -0400, John Snow wrote:

On 5/25/21 3:36 PM, Cleber Rosa wrote:

On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:

Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build.

Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
.mypy_cache).

Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).

Signed-off-by: John Snow 
---
   python/.gitignore | 19 +++
   1 file changed, 19 insertions(+)
   create mode 100644 python/.gitignore

diff --git a/python/.gitignore b/python/.gitignore
new file mode 100644
index 000..e27c99e009c
--- /dev/null
+++ b/python/.gitignore
@@ -0,0 +1,19 @@
+# python bytecode cache
+*.pyc


This is a duplicate from the parent .gitignore, so I would avoid it.


+__pycache__/


And this one is interesting because, the only thing that *should* be
in __pycache__ dirs is .pyc files (covered by the parent .gitignore
file).

So, I get the same behavior without these two entries here, so I would
skip them.  Let me know if you have any reason for explicitly
including them.

- Cleber.



Hm, not really ... Just completeness, I suppose, since this directory is
becoming increasingly separate from the rest of the tree.

It isn't crucial, it just seemed like a weird omission if they weren't
listed here. *shrug*

--js


And still, this dir is part of the overall tree.


For now 😇

   Honestly, without

any change in behavior, I'd *not* add those two ignore rules.



If you insist.

--js

Re: [RFC PATCH] configure: Do not add --warn-common to the linker flags anymore

[Richard Henderson]

On 5/25/21 4:11 AM, Thomas Huth wrote:

We are compiling with -fno-common since commit 4c288acbd6 ("configure:
Always build with -fno-common"), so --warn-common (which had been added
in commit 49237acdb725e in 2008 already) should not be necessary anymore
nowadays.

Signed-off-by: Thomas Huth
---
  Marked as RFC since I'm not 100% sure whether I'm missing here something...
  but IMHO the --warn-common does not buy us anything if we also compile
  with -fno-common...


I agree.
Reviewed-by: Richard Henderson 

r~

Re: [PATCH 2/3] meson: List if X11 dependency is detected

[Richard Henderson]

On 5/25/21 3:46 AM, Philippe Mathieu-Daudé wrote:

It is sometimes useful to know if X11 is detected.

Signed-off-by: Philippe Mathieu-Daudé
---
  meson.build | 1 +
  1 file changed, 1 insertion(+)


Reviewed-by: Richard Henderson 

r~

Re: [PATCH 1/3] meson: Only build virtfs-proxy-helper if all requisites are found

[Richard Henderson]

On 5/25/21 3:46 AM, Philippe Mathieu-Daudé wrote:

We first set have_virtfs_proxy_helper depending on have_virtfs,
then update have_virtfs... While this might work, it is not clear
when looking at the code logic. Move the have_virtfs_proxy_helper
assignation*after*  updating have_virtfs to make it obvious.

Signed-off-by: Philippe Mathieu-Daudé
---
  meson.build | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)


Reviewed-by: Richard Henderson 

r~

Re: [PATCH] replay: fix watchpoint processing for reverse debugging

[Richard Henderson]

On 5/11/21 2:11 AM, Pavel Dovgalyuk wrote:

This patch enables reverse debugging with watchpoints.
Reverse continue scans the execution to find the breakpoints
and watchpoints that should fire. It uses helper function
replay_breakpoint() for that. But this function needs to access
icount, which can't be correct in the middle of TB.
Therefore, in case of watchpoint, we have to retranslate the block
to allow this access.

Signed-off-by: Pavel Dovgalyuk
---
  softmmu/physmem.c |   10 ++
  1 file changed, 10 insertions(+)


Queued to tcg-next, thanks.


r~

[PATCH v2 11/12] linux-user/aarch64: Enable hwcap bits for bfloat16

[Richard Henderson]

Signed-off-by: Richard Henderson 
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/linux-user/elfload.c b/linux-user/elfload.c
index 1ab97e38e0..17ab06f612 100644
--- a/linux-user/elfload.c
+++ b/linux-user/elfload.c
@@ -659,7 +659,9 @@ static uint32_t get_elf_hwcap2(void)
 GET_FEATURE_ID(aa64_sve_i8mm, ARM_HWCAP2_A64_SVEI8MM);
 GET_FEATURE_ID(aa64_sve_f32mm, ARM_HWCAP2_A64_SVEF32MM);
 GET_FEATURE_ID(aa64_sve_f64mm, ARM_HWCAP2_A64_SVEF64MM);
+GET_FEATURE_ID(aa64_sve_bf16, ARM_HWCAP2_A64_SVEBF16);
 GET_FEATURE_ID(aa64_i8mm, ARM_HWCAP2_A64_I8MM);
+GET_FEATURE_ID(aa64_bf16, ARM_HWCAP2_A64_BF16);
 GET_FEATURE_ID(aa64_rndr, ARM_HWCAP2_A64_RNG);
 GET_FEATURE_ID(aa64_bti, ARM_HWCAP2_A64_BTI);
 GET_FEATURE_ID(aa64_mte, ARM_HWCAP2_A64_MTE);
-- 
2.25.1

[PATCH v2 10/12] target/arm: Implement bfloat widening fma (indexed)

[Richard Henderson]

This is BFMLAL{B,T} for both AArch64 AdvSIMD and SVE,
and VFMA{B,T}.BF16 for AArch32 NEON.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/helper.h   |  2 ++
 target/arm/neon-shared.decode |  2 ++
 target/arm/sve.decode |  2 ++
 target/arm/translate-a64.c| 15 ++-
 target/arm/translate-neon.c   | 10 ++
 target/arm/translate-sve.c| 30 ++
 target/arm/vec_helper.c   | 22 ++
 7 files changed, 82 insertions(+), 1 deletion(-)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index 36b3c9dd2d..dc6eb96d43 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -1012,6 +1012,8 @@ DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG,
 
 DEF_HELPER_FLAGS_6(gvec_bfmlal, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, ptr, i32)
+DEF_HELPER_FLAGS_6(gvec_bfmlal_idx, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, ptr, i32)
 
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode
index b61addd98b..df80e6ebf6 100644
--- a/target/arm/neon-shared.decode
+++ b/target/arm/neon-shared.decode
@@ -95,3 +95,5 @@ VFML_scalar 1110 0 . 0 s:1   1000 . 0 . 1 
index:1 ... \
rm=%vfml_scalar_q0_rm vn=%vn_sp vd=%vd_dp q=0
 VFML_scalar 1110 0 . 0 s:1   1000 . 1 . 1 . rm:3 \
index=%vfml_scalar_q1_index vn=%vn_dp vd=%vd_dp q=1
+VFMA_b16_scal   1110 0.11   1000 . q:1 . 1 . vm:3 \
+   index=%vfml_scalar_q1_index vn=%vn_dp vd=%vd_dp
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index 5281164eea..a62c169f1a 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1638,6 +1638,8 @@ FMLALB_zzxw 01100100 10 1 . 0100.0 . .
 @rrxr_3a esz=2
 FMLALT_zzxw 01100100 10 1 . 0100.1 . . @rrxr_3a esz=2
 FMLSLB_zzxw 01100100 10 1 . 0110.0 . . @rrxr_3a esz=2
 FMLSLT_zzxw 01100100 10 1 . 0110.1 . . @rrxr_3a esz=2
+BFMLALB_zzxw01100100 11 1 . 0100.0 . . @rrxr_3a esz=2
+BFMLALT_zzxw01100100 11 1 . 0100.1 . . @rrxr_3a esz=2
 
 ### SVE2 floating-point bfloat16 dot-product (indexed)
 BFDOT_zzxz  01100100 01 1 . 01 . . @rrxr_2 esz=2
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 2a99e015ca..d1dc9401d5 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -13465,18 +13465,27 @@ static void disas_simd_indexed(DisasContext *s, 
uint32_t insn)
 unallocated_encoding(s);
 return;
 }
+size = MO_32;
 break;
 case 1: /* BFDOT */
 if (is_scalar || !dc_isar_feature(aa64_bf16, s)) {
 unallocated_encoding(s);
 return;
 }
+size = MO_32;
+break;
+case 3: /* BFMLAL{B,T} */
+if (is_scalar || !dc_isar_feature(aa64_bf16, s)) {
+unallocated_encoding(s);
+return;
+}
+/* can't set is_fp without other incorrect size checks */
+size = MO_16;
 break;
 default:
 unallocated_encoding(s);
 return;
 }
-size = MO_32;
 break;
 case 0x11: /* FCMLA #0 */
 case 0x13: /* FCMLA #90 */
@@ -13606,6 +13615,10 @@ static void disas_simd_indexed(DisasContext *s, 
uint32_t insn)
 gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index,
  gen_helper_gvec_usdot_idx_b);
 return;
+case 3: /* BFMLAL{B,T} */
+gen_gvec_op4_fpst(s, 1, rd, rn, rm, rd, 0, (index << 1) | is_q,
+  gen_helper_gvec_bfmlal_idx);
+return;
 }
 g_assert_not_reached();
 case 0x11: /* FCMLA #0 */
diff --git a/target/arm/translate-neon.c b/target/arm/translate-neon.c
index 4d0c2494dc..633fef3bf7 100644
--- a/target/arm/translate-neon.c
+++ b/target/arm/translate-neon.c
@@ -4144,3 +4144,13 @@ static bool trans_VFMA_b16(DisasContext *s, arg_VFMA_b16 
*a)
 return do_neon_ddda_fpst(s, 7, a->vd, a->vn, a->vm, a->q, FPST_STD,
  gen_helper_gvec_bfmlal);
 }
+
+static bool trans_VFMA_b16_scal(DisasContext *s, arg_VFMA_b16_scal *a)
+{
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+return do_neon_ddda_fpst(s, 6, a->vd, a->vn, a->vm,
+ (a->index << 1) | a->q, FPST_STD,
+ gen_helper_gvec_bfmlal_idx);
+}
diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c
index ba8f5d7b7d..46210eb696 100644
--- a/target/arm/translate-sve.c
+++ b/target/arm/translate-sve.c
@@ -8719,3 +8719,33 @@ static bool trans_BFMLALT_zzzw(DisasContext *s, 
arg__esz *a)
 {
 return do_BFMLAL_

[PATCH v2 12/12] target/arm: Enable BFloat16 extensions

[Richard Henderson]

Disable BF16 again for !have_neon and !have_vfp during realize.

Signed-off-by: Richard Henderson 
---
 target/arm/cpu.c | 3 +++
 target/arm/cpu64.c   | 3 +++
 target/arm/cpu_tcg.c | 1 +
 3 files changed, 7 insertions(+)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 7aeb4b1381..cfc03c550b 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -1463,6 +1463,7 @@ static void arm_cpu_realizefn(DeviceState *dev, Error 
**errp)
 
 u = cpu->isar.id_isar6;
 u = FIELD_DP32(u, ID_ISAR6, JSCVT, 0);
+u = FIELD_DP32(u, ID_ISAR6, BF16, 0);
 cpu->isar.id_isar6 = u;
 
 u = cpu->isar.mvfr0;
@@ -1503,6 +1504,7 @@ static void arm_cpu_realizefn(DeviceState *dev, Error 
**errp)
 
 t = cpu->isar.id_aa64isar1;
 t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 0);
+t = FIELD_DP64(t, ID_AA64ISAR1, BF16, 0);
 t = FIELD_DP64(t, ID_AA64ISAR1, I8MM, 0);
 cpu->isar.id_aa64isar1 = t;
 
@@ -1518,6 +1520,7 @@ static void arm_cpu_realizefn(DeviceState *dev, Error 
**errp)
 u = cpu->isar.id_isar6;
 u = FIELD_DP32(u, ID_ISAR6, DP, 0);
 u = FIELD_DP32(u, ID_ISAR6, FHM, 0);
+u = FIELD_DP32(u, ID_ISAR6, BF16, 0);
 u = FIELD_DP32(u, ID_ISAR6, I8MM, 0);
 cpu->isar.id_isar6 = u;
 
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index d561dc7acc..1c23187d1a 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -661,6 +661,7 @@ static void aarch64_max_initfn(Object *obj)
 t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1);
 t = FIELD_DP64(t, ID_AA64ISAR1, SB, 1);
 t = FIELD_DP64(t, ID_AA64ISAR1, SPECRES, 1);
+t = FIELD_DP64(t, ID_AA64ISAR1, BF16, 1);
 t = FIELD_DP64(t, ID_AA64ISAR1, FRINTTS, 1);
 t = FIELD_DP64(t, ID_AA64ISAR1, LRCPC, 2); /* ARMv8.4-RCPC */
 t = FIELD_DP64(t, ID_AA64ISAR1, I8MM, 1);
@@ -708,6 +709,7 @@ static void aarch64_max_initfn(Object *obj)
 t = FIELD_DP64(t, ID_AA64ZFR0, SVEVER, 1);
 t = FIELD_DP64(t, ID_AA64ZFR0, AES, 2);  /* PMULL */
 t = FIELD_DP64(t, ID_AA64ZFR0, BITPERM, 1);
+t = FIELD_DP64(t, ID_AA64ZFR0, BFLOAT16, 1);
 t = FIELD_DP64(t, ID_AA64ZFR0, SHA3, 1);
 t = FIELD_DP64(t, ID_AA64ZFR0, SM4, 1);
 t = FIELD_DP64(t, ID_AA64ZFR0, I8MM, 1);
@@ -731,6 +733,7 @@ static void aarch64_max_initfn(Object *obj)
 u = FIELD_DP32(u, ID_ISAR6, FHM, 1);
 u = FIELD_DP32(u, ID_ISAR6, SB, 1);
 u = FIELD_DP32(u, ID_ISAR6, SPECRES, 1);
+u = FIELD_DP32(u, ID_ISAR6, BF16, 1);
 u = FIELD_DP32(u, ID_ISAR6, I8MM, 1);
 cpu->isar.id_isar6 = u;
 
diff --git a/target/arm/cpu_tcg.c b/target/arm/cpu_tcg.c
index d3458335ed..c8a12bc2d6 100644
--- a/target/arm/cpu_tcg.c
+++ b/target/arm/cpu_tcg.c
@@ -968,6 +968,7 @@ static void arm_max_initfn(Object *obj)
 t = FIELD_DP32(t, ID_ISAR6, FHM, 1);
 t = FIELD_DP32(t, ID_ISAR6, SB, 1);
 t = FIELD_DP32(t, ID_ISAR6, SPECRES, 1);
+t = FIELD_DP32(t, ID_ISAR6, BF16, 1);
 t = FIELD_DP32(t, ID_ISAR6, I8MM, 1);
 cpu->isar.id_isar6 = t;
 
-- 
2.25.1

[PATCH v2 06/12] target/arm: Implement bfloat16 dot product (vector)

[Richard Henderson]

This is BFDOT for both AArch64 AdvSIMD and SVE,
and VDOT.BF16 for AArch32 NEON.

Signed-off-by: Richard Henderson 
---
 target/arm/helper.h   |  3 +++
 target/arm/neon-shared.decode |  2 ++
 target/arm/sve.decode |  3 +++
 target/arm/translate-a64.c| 20 ++
 target/arm/translate-neon.c   |  9 
 target/arm/translate-sve.c| 12 +++
 target/arm/vec_helper.c   | 40 +++
 7 files changed, 89 insertions(+)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index 8b4b7d92f3..de2f5331dc 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -1002,6 +1002,9 @@ DEF_HELPER_FLAGS_5(gvec_ummla_b, TCG_CALL_NO_RWG,
 DEF_HELPER_FLAGS_5(gvec_usmmla_b, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
 
+DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, i32)
+
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
 #include "helper-sve.h"
diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode
index cc9f4cdd85..31a0839bbb 100644
--- a/target/arm/neon-shared.decode
+++ b/target/arm/neon-shared.decode
@@ -52,6 +52,8 @@ VUDOT   110 00 . 10   1101 . q:1 . 1  
\
vm=%vm_dp vn=%vn_dp vd=%vd_dp
 VUSDOT  110 01 . 10   1101 . q:1 . 0  \
vm=%vm_dp vn=%vn_dp vd=%vd_dp
+VDOT_b16    110 00 . 00   1101 . q:1 . 0  \
+   vm=%vm_dp vn=%vn_dp vd=%vd_dp
 
 # VFM[AS]L
 VFML    110 0 s:1 . 10   1000 . 0 . 1  \
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index 18d1a0eecc..a7429b293f 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1625,6 +1625,9 @@ FMLALT_zzzw 01100100 10 1 . 10 0 00 1 . . 
 @rda_rn_rm_e0
 FMLSLB_zzzw 01100100 10 1 . 10 1 00 0 . .  @rda_rn_rm_e0
 FMLSLT_zzzw 01100100 10 1 . 10 1 00 1 . .  @rda_rn_rm_e0
 
+### SVE2 floating-point bfloat16 dot-product
+BFDOT_  01100100 01 1 . 10 0 00 0 . .  @rda_rn_rm_e0
+
 ### SVE2 floating-point multiply-add long (indexed)
 FMLALB_zzxw 01100100 10 1 . 0100.0 . . @rrxr_3a esz=2
 FMLALT_zzxw 01100100 10 1 . 0100.1 . . @rrxr_3a esz=2
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 5a96523b9f..127c8d8e9d 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -12228,6 +12228,16 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 }
 feature = dc_isar_feature(aa64_fcma, s);
 break;
+case 0x1f: /* BFDOT */
+switch (size) {
+case 1:
+feature = dc_isar_feature(aa64_bf16, s);
+break;
+default:
+unallocated_encoding(s);
+return;
+}
+break;
 default:
 unallocated_encoding(s);
 return;
@@ -12311,6 +12321,16 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 }
 return;
 
+case 0xf: /* BFDOT */
+switch (size) {
+case 1:
+gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, 
gen_helper_gvec_bfdot);
+break;
+default:
+g_assert_not_reached();
+}
+return;
+
 default:
 g_assert_not_reached();
 }
diff --git a/target/arm/translate-neon.c b/target/arm/translate-neon.c
index 6d94229c69..9460857b2a 100644
--- a/target/arm/translate-neon.c
+++ b/target/arm/translate-neon.c
@@ -296,6 +296,15 @@ static bool trans_VUSDOT(DisasContext *s, arg_VUSDOT *a)
 gen_helper_gvec_usdot_b);
 }
 
+static bool trans_VDOT_b16(DisasContext *s, arg_VDOT_b16 *a)
+{
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+return do_neon_ddda(s, a->q * 7, a->vd, a->vn, a->vm, 0,
+gen_helper_gvec_bfdot);
+}
+
 static bool trans_VFML(DisasContext *s, arg_VFML *a)
 {
 int opr_sz;
diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c
index fb692a1835..ed290827ad 100644
--- a/target/arm/translate-sve.c
+++ b/target/arm/translate-sve.c
@@ -8653,3 +8653,15 @@ static bool trans_UMMLA(DisasContext *s, arg__esz *a)
 {
 return do_i8mm__ool(s, a, gen_helper_gvec_ummla_b, 0);
 }
+
+static bool trans_BFDOT_(DisasContext *s, arg__esz *a)
+{
+if (!dc_isar_feature(aa64_sve_bf16, s)) {
+return false;
+}
+if (sve_access_check(s)) {
+gen_gvec_ool_(s, gen_helper_gvec_bfdot,
+  a->rd, a->rn, a->rm, a->ra, 0);
+}
+return true;
+}
diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c
index e84b438340..7eefcd06ea 100644
--- a/target/arm/vec_helper.c
+++ b/target/arm/vec_helper.c
@@ -2412,3 +2412,43 @@ static void do_mmla_b(void *vd, void *vn, void *vm, void 
*va, uint32_t desc,
 DO_MMLA_B(gvec_smmla_b, do_smm

[PATCH v2 07/12] target/arm: Implement bfloat16 dot product (indexed)

[Richard Henderson]

This is BFDOT for both AArch64 AdvSIMD and SVE,
and VDOT.BF16 for AArch32 NEON.

Signed-off-by: Richard Henderson 
---
 target/arm/helper.h   |  2 ++
 target/arm/neon-shared.decode |  2 ++
 target/arm/sve.decode |  3 +++
 target/arm/translate-a64.c| 41 +++
 target/arm/translate-neon.c   |  9 
 target/arm/translate-sve.c| 12 ++
 target/arm/vec_helper.c   | 20 +
 7 files changed, 80 insertions(+), 9 deletions(-)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index de2f5331dc..376c1cef0f 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -1004,6 +1004,8 @@ DEF_HELPER_FLAGS_5(gvec_usmmla_b, TCG_CALL_NO_RWG,
 
 DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
+DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, i32)
 
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode
index 31a0839bbb..fa3cf14e3a 100644
--- a/target/arm/neon-shared.decode
+++ b/target/arm/neon-shared.decode
@@ -81,6 +81,8 @@ VUSDOT_scalar   1110 1 . 00   1101 . q:1 index:1 
0 vm:4 \
vn=%vn_dp vd=%vd_dp
 VSUDOT_scalar   1110 1 . 00   1101 . q:1 index:1 1 vm:4 \
vn=%vn_dp vd=%vd_dp
+VDOT_b16_scal   1110 0 . 00   1101 . q:1 index:1 0 vm:4 \
+   vn=%vn_dp vd=%vd_dp
 
 %vfml_scalar_q0_rm 0:3 5:1
 %vfml_scalar_q1_index 5:1 3:1
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index a7429b293f..51f87e8937 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1633,3 +1633,6 @@ FMLALB_zzxw 01100100 10 1 . 0100.0 . .
 @rrxr_3a esz=2
 FMLALT_zzxw 01100100 10 1 . 0100.1 . . @rrxr_3a esz=2
 FMLSLB_zzxw 01100100 10 1 . 0110.0 . . @rrxr_3a esz=2
 FMLSLT_zzxw 01100100 10 1 . 0110.1 . . @rrxr_3a esz=2
+
+### SVE2 floating-point bfloat16 dot-product (indexed)
+BFDOT_zzxz  01100100 01 1 . 01 . . @rrxr_2 esz=2
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 127c8d8e9d..1df931cef5 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -13442,8 +13442,22 @@ static void disas_simd_indexed(DisasContext *s, 
uint32_t insn)
 return;
 }
 break;
-case 0x0f: /* SUDOT, USDOT */
-if (is_scalar || (size & 1) || !dc_isar_feature(aa64_i8mm, s)) {
+case 0x0f:
+switch (size) {
+case 0: /* SUDOT */
+case 2: /* USDOT */
+if (is_scalar || !dc_isar_feature(aa64_i8mm, s)) {
+unallocated_encoding(s);
+return;
+}
+break;
+case 1: /* BFDOT */
+if (is_scalar || !dc_isar_feature(aa64_bf16, s)) {
+unallocated_encoding(s);
+return;
+}
+break;
+default:
 unallocated_encoding(s);
 return;
 }
@@ -13563,13 +13577,22 @@ static void disas_simd_indexed(DisasContext *s, 
uint32_t insn)
  u ? gen_helper_gvec_udot_idx_b
  : gen_helper_gvec_sdot_idx_b);
 return;
-case 0x0f: /* SUDOT, USDOT */
-gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index,
- extract32(insn, 23, 1)
- ? gen_helper_gvec_usdot_idx_b
- : gen_helper_gvec_sudot_idx_b);
-return;
-
+case 0x0f:
+switch (extract32(insn, 22, 2)) {
+case 0: /* SUDOT */
+gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index,
+ gen_helper_gvec_sudot_idx_b);
+return;
+case 1: /* BFDOT */
+gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index,
+ gen_helper_gvec_bfdot_idx);
+return;
+case 2: /* USDOT */
+gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, index,
+ gen_helper_gvec_usdot_idx_b);
+return;
+}
+g_assert_not_reached();
 case 0x11: /* FCMLA #0 */
 case 0x13: /* FCMLA #90 */
 case 0x15: /* FCMLA #180 */
diff --git a/target/arm/translate-neon.c b/target/arm/translate-neon.c
index 9460857b2a..8099767792 100644
--- a/target/arm/translate-neon.c
+++ b/target/arm/translate-neon.c
@@ -390,6 +390,15 @@ static bool trans_VSUDOT_scalar(DisasContext *s, 
arg_VSUDOT_scalar *a)
 gen_helper_gvec_sudot_idx_b);
 }
 
+static bool trans_VDOT_b16_scal(DisasContext *s, arg_VDOT_b16_scal *a)
+{
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+return do_neon_ddda(s, a->q * 6, a->vd, a->vn, a->vm, a->index,
+gen_helper_gvec_bfdot_idx);
+}
+
 static bool trans_VFML_scalar(DisasContext *s, arg_VFM

[PATCH v2 08/12] target/arm: Implement bfloat16 matrix multiply accumulate

[Richard Henderson]

This is BFMMLA for both AArch64 AdvSIMD and SVE,
and VMMLA.BF16 for AArch32 NEON.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/helper.h   |  3 +++
 target/arm/neon-shared.decode |  2 ++
 target/arm/sve.decode |  6 +++--
 target/arm/translate-a64.c| 10 +
 target/arm/translate-neon.c   |  9 
 target/arm/translate-sve.c| 12 ++
 target/arm/vec_helper.c   | 42 ++-
 7 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index 376c1cef0f..af75d7f25f 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -1007,6 +1007,9 @@ DEF_HELPER_FLAGS_5(gvec_bfdot, TCG_CALL_NO_RWG,
 DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
 
+DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, i32)
+
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
 #include "helper-sve.h"
diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode
index fa3cf14e3a..4e0a25d27c 100644
--- a/target/arm/neon-shared.decode
+++ b/target/arm/neon-shared.decode
@@ -67,6 +67,8 @@ VUMMLA  1100 0.10   1100 .1.1  \
vm=%vm_dp vn=%vn_dp vd=%vd_dp
 VUSMMLA 1100 1.10   1100 .1.0  \
vm=%vm_dp vn=%vn_dp vd=%vd_dp
+VMMLA_b16   1100 0.00   1100 .1.0  \
+   vm=%vm_dp vn=%vn_dp vd=%vd_dp
 
 VCMLA_scalar    1110 0 . rot:2   1000 . q:1 index:1 0 vm:4 \
vn=%vn_dp vd=%vd_dp size=1
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index 51f87e8937..6c17898dee 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1568,8 +1568,10 @@ SQRDCMLAH_  01000100 esz:2 0 rm:5 0011 rot:2 rn:5 
rd:5  ra=%reg_movprfx
 USDOT_  01000100 .. 0 . 011 110 . .  @rda_rn_rm
 
 ### SVE2 floating point matrix multiply accumulate
-
-FMMLA   01100100 .. 1 . 111001 . .  @rda_rn_rm
+{
+  BFMMLA01100100 01 1 . 111 001 . .  @rda_rn_rm_e0
+  FMMLA 01100100 .. 1 . 111 001 . .  @rda_rn_rm
+}
 
 ### SVE2 Memory Gather Load Group
 
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 1df931cef5..9f8dae90ba 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -12228,6 +12228,13 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 }
 feature = dc_isar_feature(aa64_fcma, s);
 break;
+case 0x1d: /* BFMMLA */
+if (size != MO_16 || !is_q) {
+unallocated_encoding(s);
+return;
+}
+feature = dc_isar_feature(aa64_bf16, s);
+break;
 case 0x1f: /* BFDOT */
 switch (size) {
 case 1:
@@ -12321,6 +12328,9 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 }
 return;
 
+case 0xd: /* BFMMLA */
+gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfmmla);
+return;
 case 0xf: /* BFDOT */
 switch (size) {
 case 1:
diff --git a/target/arm/translate-neon.c b/target/arm/translate-neon.c
index 8099767792..9d227a1e13 100644
--- a/target/arm/translate-neon.c
+++ b/target/arm/translate-neon.c
@@ -4126,3 +4126,12 @@ static bool trans_VUSMMLA(DisasContext *s, arg_VUSMMLA 
*a)
 return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0,
 gen_helper_gvec_usmmla_b);
 }
+
+static bool trans_VMMLA_b16(DisasContext *s, arg_VMMLA_b16 *a)
+{
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0,
+gen_helper_gvec_bfmmla);
+}
diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c
index 6f02030635..4f575dc334 100644
--- a/target/arm/translate-sve.c
+++ b/target/arm/translate-sve.c
@@ -8677,3 +8677,15 @@ static bool trans_BFDOT_zzxz(DisasContext *s, 
arg_rrxr_esz *a)
 }
 return true;
 }
+
+static bool trans_BFMMLA(DisasContext *s, arg__esz *a)
+{
+if (!dc_isar_feature(aa64_sve_bf16, s)) {
+return false;
+}
+if (sve_access_check(s)) {
+gen_gvec_ool_(s, gen_helper_gvec_bfmmla,
+  a->rd, a->rn, a->rm, a->ra, 0);
+}
+return true;
+}
diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c
index 74a497f38c..27e9bdd329 100644
--- a/target/arm/vec_helper.c
+++ b/target/arm/vec_helper.c
@@ -2385,7 +2385,7 @@ static void do_mmla_b(void *vd, void *vn, void *vm, void 
*va, uint32_t desc,
  * Process the entire segment at once, writing back the
  * results only after we've consumed all of the inputs.
  *
- * Key to indicies by column:
+ * Key to indices by column:
  *  i   j  i j
  

[PATCH v2 09/12] target/arm: Implement bfloat widening fma (vector)

[Richard Henderson]

This is BFMLAL{B,T} for both AArch64 AdvSIMD and SVE,
and VFMA{B,T}.BF16 for AArch32 NEON.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/helper.h   |  3 +++
 target/arm/neon-shared.decode |  3 +++
 target/arm/sve.decode |  3 +++
 target/arm/translate-a64.c| 13 +
 target/arm/translate-neon.c   |  9 +
 target/arm/translate-sve.c| 30 ++
 target/arm/vec_helper.c   | 16 
 7 files changed, 73 insertions(+), 4 deletions(-)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index af75d7f25f..36b3c9dd2d 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -1010,6 +1010,9 @@ DEF_HELPER_FLAGS_5(gvec_bfdot_idx, TCG_CALL_NO_RWG,
 DEF_HELPER_FLAGS_5(gvec_bfmmla, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
 
+DEF_HELPER_FLAGS_6(gvec_bfmlal, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, ptr, i32)
+
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
 #include "helper-sve.h"
diff --git a/target/arm/neon-shared.decode b/target/arm/neon-shared.decode
index 4e0a25d27c..b61addd98b 100644
--- a/target/arm/neon-shared.decode
+++ b/target/arm/neon-shared.decode
@@ -70,6 +70,9 @@ VUSMMLA 1100 1.10   1100 .1.0  \
 VMMLA_b16   1100 0.00   1100 .1.0  \
vm=%vm_dp vn=%vn_dp vd=%vd_dp
 
+VFMA_b16    110 0 0.11   1000 . q:1 . 1  \
+   vm=%vm_dp vn=%vn_dp vd=%vd_dp
+
 VCMLA_scalar    1110 0 . rot:2   1000 . q:1 index:1 0 vm:4 \
vn=%vn_dp vd=%vd_dp size=1
 VCMLA_scalar    1110 1 . rot:2   1000 . q:1 . 0  \
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index 6c17898dee..5281164eea 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1627,6 +1627,9 @@ FMLALT_zzzw 01100100 10 1 . 10 0 00 1 . . 
 @rda_rn_rm_e0
 FMLSLB_zzzw 01100100 10 1 . 10 1 00 0 . .  @rda_rn_rm_e0
 FMLSLT_zzzw 01100100 10 1 . 10 1 00 1 . .  @rda_rn_rm_e0
 
+BFMLALB_zzzw01100100 11 1 . 10 0 00 0 . .  @rda_rn_rm_e0
+BFMLALT_zzzw01100100 11 1 . 10 0 00 1 . .  @rda_rn_rm_e0
+
 ### SVE2 floating-point bfloat16 dot-product
 BFDOT_  01100100 01 1 . 10 0 00 0 . .  @rda_rn_rm_e0
 
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 9f8dae90ba..2a99e015ca 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -12235,9 +12235,10 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 }
 feature = dc_isar_feature(aa64_bf16, s);
 break;
-case 0x1f: /* BFDOT */
+case 0x1f:
 switch (size) {
-case 1:
+case 1: /* BFDOT */
+case 3: /* BFMLAL{B,T} */
 feature = dc_isar_feature(aa64_bf16, s);
 break;
 default:
@@ -12331,11 +12332,15 @@ static void 
disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn)
 case 0xd: /* BFMMLA */
 gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, gen_helper_gvec_bfmmla);
 return;
-case 0xf: /* BFDOT */
+case 0xf:
 switch (size) {
-case 1:
+case 1: /* BFDOT */
 gen_gvec_op4_ool(s, is_q, rd, rn, rm, rd, 0, 
gen_helper_gvec_bfdot);
 break;
+case 3: /* BFMLAL{B,T} */
+gen_gvec_op4_fpst(s, 1, rd, rn, rm, rd, false, is_q,
+  gen_helper_gvec_bfmlal);
+break;
 default:
 g_assert_not_reached();
 }
diff --git a/target/arm/translate-neon.c b/target/arm/translate-neon.c
index 9d227a1e13..4d0c2494dc 100644
--- a/target/arm/translate-neon.c
+++ b/target/arm/translate-neon.c
@@ -4135,3 +4135,12 @@ static bool trans_VMMLA_b16(DisasContext *s, 
arg_VMMLA_b16 *a)
 return do_neon_ddda(s, 7, a->vd, a->vn, a->vm, 0,
 gen_helper_gvec_bfmmla);
 }
+
+static bool trans_VFMA_b16(DisasContext *s, arg_VFMA_b16 *a)
+{
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+return do_neon_ddda_fpst(s, 7, a->vd, a->vn, a->vm, a->q, FPST_STD,
+ gen_helper_gvec_bfmlal);
+}
diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c
index 4f575dc334..ba8f5d7b7d 100644
--- a/target/arm/translate-sve.c
+++ b/target/arm/translate-sve.c
@@ -8689,3 +8689,33 @@ static bool trans_BFMMLA(DisasContext *s, arg__esz 
*a)
 }
 return true;
 }
+
+static bool do_BFMLAL_zzzw(DisasContext *s, arg__esz *a, bool sel)
+{
+if (!dc_isar_feature(aa64_sve_bf16, s)) {
+return false;
+}
+if (sve_access_check(s)) {
+TCGv_ptr status = fpstatus_ptr(FPST_FPCR);
+unsigned vsz = vec_full_reg_size(s);
+
+tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd),
+   vec_full_reg_offset(s, a->rn),
+

[PATCH v2 05/12] softfpu: Add float_round_to_odd_inf

[Richard Henderson]

For Arm BFDOT and BFMMLA, we need a version of round-to-odd
that overflows to infinity, instead of the max normal number.

Cc: Alex Bennée 
Signed-off-by: Richard Henderson 
---
 include/fpu/softfloat-types.h | 4 +++-
 fpu/softfloat-parts.c.inc | 6 --
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/include/fpu/softfloat-types.h b/include/fpu/softfloat-types.h
index 8a3f20fae9..3b757c3d6a 100644
--- a/include/fpu/softfloat-types.h
+++ b/include/fpu/softfloat-types.h
@@ -134,8 +134,10 @@ typedef enum __attribute__((__packed__)) {
 float_round_up   = 2,
 float_round_to_zero  = 3,
 float_round_ties_away= 4,
-/* Not an IEEE rounding mode: round to the closest odd mantissa value */
+/* Not an IEEE rounding mode: round to closest odd, overflow to max */
 float_round_to_odd   = 5,
+/* Not an IEEE rounding mode: round to closest odd, overflow to inf */
+float_round_to_odd_inf   = 6,
 } FloatRoundMode;
 
 /*
diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc
index a897a5a743..7f69da1d8f 100644
--- a/fpu/softfloat-parts.c.inc
+++ b/fpu/softfloat-parts.c.inc
@@ -176,13 +176,12 @@ static void partsN(uncanon)(FloatPartsN *p, float_status 
*s,
 g_assert_not_reached();
 }
 
+overflow_norm = false;
 switch (s->float_rounding_mode) {
 case float_round_nearest_even:
-overflow_norm = false;
 inc = ((p->frac_lo & roundeven_mask) != frac_lsbm1 ? frac_lsbm1 : 0);
 break;
 case float_round_ties_away:
-overflow_norm = false;
 inc = frac_lsbm1;
 break;
 case float_round_to_zero:
@@ -199,6 +198,8 @@ static void partsN(uncanon)(FloatPartsN *p, float_status *s,
 break;
 case float_round_to_odd:
 overflow_norm = true;
+/* fall through */
+case float_round_to_odd_inf:
 inc = p->frac_lo & frac_lsb ? 0 : round_mask;
 break;
 default:
@@ -259,6 +260,7 @@ static void partsN(uncanon)(FloatPartsN *p, float_status *s,
? frac_lsbm1 : 0);
 break;
 case float_round_to_odd:
+case float_round_to_odd_inf:
 inc = p->frac_lo & frac_lsb ? 0 : round_mask;
 break;
 default:
-- 
2.25.1

[PATCH v2 01/12] target/arm: Add isar_feature_{aa32, aa64, aa64_sve}_bf16

[Richard Henderson]

Note that the SVE BFLOAT16 support does not require SVE2,
it is an independent extension.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/cpu.h | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index 04f8be35bf..d68275b15e 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -3783,6 +3783,11 @@ static inline bool isar_feature_aa32_predinv(const 
ARMISARegisters *id)
 return FIELD_EX32(id->id_isar6, ID_ISAR6, SPECRES) != 0;
 }
 
+static inline bool isar_feature_aa32_bf16(const ARMISARegisters *id)
+{
+return FIELD_EX32(id->id_isar6, ID_ISAR6, BF16) != 0;
+}
+
 static inline bool isar_feature_aa32_i8mm(const ARMISARegisters *id)
 {
 return FIELD_EX32(id->id_isar6, ID_ISAR6, I8MM) != 0;
@@ -4122,6 +4127,11 @@ static inline bool isar_feature_aa64_dcpodp(const 
ARMISARegisters *id)
 return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, DPB) >= 2;
 }
 
+static inline bool isar_feature_aa64_bf16(const ARMISARegisters *id)
+{
+return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, BF16) != 0;
+}
+
 static inline bool isar_feature_aa64_fp_simd(const ARMISARegisters *id)
 {
 /* We always set the AdvSIMD and FP fields identically.  */
@@ -4266,6 +4276,11 @@ static inline bool isar_feature_aa64_sve2_bitperm(const 
ARMISARegisters *id)
 return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, BITPERM) != 0;
 }
 
+static inline bool isar_feature_aa64_sve_bf16(const ARMISARegisters *id)
+{
+return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, BFLOAT16) != 0;
+}
+
 static inline bool isar_feature_aa64_sve2_sha3(const ARMISARegisters *id)
 {
 return FIELD_EX64(id->id_aa64zfr0, ID_AA64ZFR0, SHA3) != 0;
-- 
2.25.1

[PATCH v2 03/12] target/arm: Implement scalar float32 to bfloat16 conversion

[Richard Henderson]

This is the 64-bit BFCVT and the 32-bit VCVT{B,T}.BF16.F32.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/helper.h|  1 +
 target/arm/vfp.decode  |  2 ++
 target/arm/translate-a64.c | 19 +++
 target/arm/translate-vfp.c | 24 
 target/arm/vfp_helper.c|  5 +
 5 files changed, 51 insertions(+)

diff --git a/target/arm/helper.h b/target/arm/helper.h
index 23ccb0f72f..9977a827e9 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -143,6 +143,7 @@ DEF_HELPER_3(vfp_cmped, void, f64, f64, env)
 
 DEF_HELPER_2(vfp_fcvtds, f64, f32, env)
 DEF_HELPER_2(vfp_fcvtsd, f32, f64, env)
+DEF_HELPER_FLAGS_2(bfcvt, TCG_CALL_NO_RWG, i32, f32, ptr)
 
 DEF_HELPER_2(vfp_uitoh, f16, i32, ptr)
 DEF_HELPER_2(vfp_uitos, f32, i32, ptr)
diff --git a/target/arm/vfp.decode b/target/arm/vfp.decode
index 6f7f28f9a4..52535d9b0b 100644
--- a/target/arm/vfp.decode
+++ b/target/arm/vfp.decode
@@ -205,6 +205,8 @@ VCVT_f64_f16  1110 1.11 0010  1011 t:1 1.0  \
 
 # VCVTB and VCVTT to f16: Vd format is always vd_sp;
 # Vm format depends on size bit
+VCVT_b16_f32  1110 1.11 0011  1001 t:1 1.0  \
+ vd=%vd_sp vm=%vm_sp
 VCVT_f16_f32  1110 1.11 0011  1010 t:1 1.0  \
  vd=%vd_sp vm=%vm_sp
 VCVT_f16_f64  1110 1.11 0011  1011 t:1 1.0  \
diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 510cb6ca5e..90605d7dce 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -6273,6 +6273,9 @@ static void handle_fp_1src_single(DisasContext *s, int 
opcode, int rd, int rn)
 case 0x3: /* FSQRT */
 gen_helper_vfp_sqrts(tcg_res, tcg_op, cpu_env);
 goto done;
+case 0x6: /* BFCVT */
+gen_fpst = gen_helper_bfcvt;
+break;
 case 0x8: /* FRINTN */
 case 0x9: /* FRINTP */
 case 0xa: /* FRINTM */
@@ -6550,6 +6553,22 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn)
 }
 break;
 
+case 0x6:
+switch (type) {
+case 1: /* BFCVT */
+if (!dc_isar_feature(aa64_bf16, s)) {
+goto do_unallocated;
+}
+if (!fp_access_check(s)) {
+return;
+}
+handle_fp_1src_single(s, opcode, rd, rn);
+break;
+default:
+goto do_unallocated;
+}
+break;
+
 default:
 do_unallocated:
 unallocated_encoding(s);
diff --git a/target/arm/translate-vfp.c b/target/arm/translate-vfp.c
index 3da84f30a0..d8271dbaac 100644
--- a/target/arm/translate-vfp.c
+++ b/target/arm/translate-vfp.c
@@ -3025,6 +3025,30 @@ static bool trans_VCVT_f64_f16(DisasContext *s, 
arg_VCVT_f64_f16 *a)
 return true;
 }
 
+static bool trans_VCVT_b16_f32(DisasContext *s, arg_VCVT_b16_f32 *a)
+{
+TCGv_ptr fpst;
+TCGv_i32 tmp;
+
+if (!dc_isar_feature(aa32_bf16, s)) {
+return false;
+}
+
+if (!vfp_access_check(s)) {
+return true;
+}
+
+fpst = fpstatus_ptr(FPST_FPCR);
+tmp = tcg_temp_new_i32();
+
+vfp_load_reg32(tmp, a->vm);
+gen_helper_bfcvt(tmp, tmp, fpst);
+tcg_gen_st16_i32(tmp, cpu_env, vfp_f16_offset(a->vd, a->t));
+tcg_temp_free_ptr(fpst);
+tcg_temp_free_i32(tmp);
+return true;
+}
+
 static bool trans_VCVT_f16_f32(DisasContext *s, arg_VCVT_f16_f32 *a)
 {
 TCGv_ptr fpst;
diff --git a/target/arm/vfp_helper.c b/target/arm/vfp_helper.c
index 01b9d8557f..fe7a2a5daa 100644
--- a/target/arm/vfp_helper.c
+++ b/target/arm/vfp_helper.c
@@ -408,6 +408,11 @@ float32 VFP_HELPER(fcvts, d)(float64 x, CPUARMState *env)
 return float64_to_float32(x, &env->vfp.fp_status);
 }
 
+uint32_t HELPER(bfcvt)(float32 x, void *status)
+{
+return float32_to_bfloat16(x, status);
+}
+
 /*
  * VFP3 fixed point conversion. The AArch32 versions of fix-to-float
  * must always round-to-nearest; the AArch64 ones honour the FPSCR
-- 
2.25.1

[PATCH v2 02/12] target/arm: Unify unallocated path in disas_fp_1src

[Richard Henderson]

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/translate-a64.c | 15 ++-
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index ceac0ee2bd..510cb6ca5e 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -6494,8 +6494,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn)
 int rd = extract32(insn, 0, 5);
 
 if (mos) {
-unallocated_encoding(s);
-return;
+goto do_unallocated;
 }
 
 switch (opcode) {
@@ -6504,8 +6503,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn)
 /* FCVT between half, single and double precision */
 int dtype = extract32(opcode, 0, 2);
 if (type == 2 || dtype == type) {
-unallocated_encoding(s);
-return;
+goto do_unallocated;
 }
 if (!fp_access_check(s)) {
 return;
@@ -6517,8 +6515,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn)
 
 case 0x10 ... 0x13: /* FRINT{32,64}{X,Z} */
 if (type > 1 || !dc_isar_feature(aa64_frint, s)) {
-unallocated_encoding(s);
-return;
+goto do_unallocated;
 }
 /* fall through */
 case 0x0 ... 0x3:
@@ -6540,8 +6537,7 @@ static void disas_fp_1src(DisasContext *s, uint32_t insn)
 break;
 case 3:
 if (!dc_isar_feature(aa64_fp16, s)) {
-unallocated_encoding(s);
-return;
+goto do_unallocated;
 }
 
 if (!fp_access_check(s)) {
@@ -6550,11 +6546,12 @@ static void disas_fp_1src(DisasContext *s, uint32_t 
insn)
 handle_fp_1src_half(s, opcode, rd, rn);
 break;
 default:
-unallocated_encoding(s);
+goto do_unallocated;
 }
 break;
 
 default:
+do_unallocated:
 unallocated_encoding(s);
 break;
 }
-- 
2.25.1

[PATCH v2 04/12] target/arm: Implement vector float32 to bfloat16 conversion

[Richard Henderson]

This is BFCVT{N,T} for both AArch64 AdvSIMD and SVE,
and VCVT.BF16.F32 for AArch32 NEON.

Reviewed-by: Peter Maydell 
Signed-off-by: Richard Henderson 
---
 target/arm/helper-sve.h |  4 
 target/arm/helper.h |  1 +
 target/arm/neon-dp.decode   |  1 +
 target/arm/sve.decode   |  2 ++
 target/arm/sve_helper.c |  2 ++
 target/arm/translate-a64.c  | 17 ++
 target/arm/translate-neon.c | 45 +
 target/arm/translate-sve.c  | 16 +
 target/arm/vfp_helper.c |  7 ++
 9 files changed, 95 insertions(+)

diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h
index 29a14a21f5..dc629f851a 100644
--- a/target/arm/helper-sve.h
+++ b/target/arm/helper-sve.h
@@ -1197,6 +1197,8 @@ DEF_HELPER_FLAGS_5(sve_fcvt_hd, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
 DEF_HELPER_FLAGS_5(sve_fcvt_sd, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
+DEF_HELPER_FLAGS_5(sve_bfcvt, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, i32)
 
 DEF_HELPER_FLAGS_5(sve_fcvtzs_hh, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
@@ -2752,6 +2754,8 @@ DEF_HELPER_FLAGS_5(sve2_fcvtnt_sh, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
 DEF_HELPER_FLAGS_5(sve2_fcvtnt_ds, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
+DEF_HELPER_FLAGS_5(sve_bfcvtnt, TCG_CALL_NO_RWG,
+   void, ptr, ptr, ptr, ptr, i32)
 
 DEF_HELPER_FLAGS_5(sve2_fcvtlt_hs, TCG_CALL_NO_RWG,
void, ptr, ptr, ptr, ptr, i32)
diff --git a/target/arm/helper.h b/target/arm/helper.h
index 9977a827e9..8b4b7d92f3 100644
--- a/target/arm/helper.h
+++ b/target/arm/helper.h
@@ -144,6 +144,7 @@ DEF_HELPER_3(vfp_cmped, void, f64, f64, env)
 DEF_HELPER_2(vfp_fcvtds, f64, f32, env)
 DEF_HELPER_2(vfp_fcvtsd, f32, f64, env)
 DEF_HELPER_FLAGS_2(bfcvt, TCG_CALL_NO_RWG, i32, f32, ptr)
+DEF_HELPER_FLAGS_2(bfcvt_pair, TCG_CALL_NO_RWG, i32, i64, ptr)
 
 DEF_HELPER_2(vfp_uitoh, f16, i32, ptr)
 DEF_HELPER_2(vfp_uitos, f32, i32, ptr)
diff --git a/target/arm/neon-dp.decode b/target/arm/neon-dp.decode
index ec83f10ab3..fd3a01bfa0 100644
--- a/target/arm/neon-dp.decode
+++ b/target/arm/neon-dp.decode
@@ -521,6 +521,7 @@ Vimm_1r   001 . 1 . 000 ...  cmode:4 0 . 
op:1 1  @1reg_imm
 VRINTZ    001 11 . 11 .. 10  0 1011 . . 0  @2misc
 
 VCVT_F16_F32  001 11 . 11 .. 10  0 1100 0 . 0  @2misc_q0
+VCVT_B16_F32  001 11 . 11 .. 10  0 1100 1 . 0  @2misc_q0
 
 VRINTM    001 11 . 11 .. 10  0 1101 . . 0  @2misc
 
diff --git a/target/arm/sve.decode b/target/arm/sve.decode
index cb077bfde9..18d1a0eecc 100644
--- a/target/arm/sve.decode
+++ b/target/arm/sve.decode
@@ -1036,6 +1036,7 @@ FNMLS_zpzzz 01100101 .. 1 . 111 ... . .   
  @rdn_pg_rm_ra
 # SVE floating-point convert precision
 FCVT_sh 01100101 10 0010 00 101 ... . . @rd_pg_rn_e0
 FCVT_hs 01100101 10 0010 01 101 ... . . @rd_pg_rn_e0
+BFCVT   01100101 10 0010 10 101 ... . . @rd_pg_rn_e0
 FCVT_dh 01100101 11 0010 00 101 ... . . @rd_pg_rn_e0
 FCVT_hd 01100101 11 0010 01 101 ... . . @rd_pg_rn_e0
 FCVT_ds 01100101 11 0010 10 101 ... . . @rd_pg_rn_e0
@@ -1610,6 +1611,7 @@ RAX101000101 00 1 . 0 1 . .  
@rd_rn_rm_e0
 FCVTXNT_ds  01100100 00 0010 10 101 ... . .  @rd_pg_rn_e0
 FCVTX_ds01100101 00 0010 10 101 ... . .  @rd_pg_rn_e0
 FCVTNT_sh   01100100 10 0010 00 101 ... . .  @rd_pg_rn_e0
+BFCVTNT 01100100 10 0010 10 101 ... . .  @rd_pg_rn_e0
 FCVTLT_hs   01100100 10 0010 01 101 ... . .  @rd_pg_rn_e0
 FCVTNT_ds   01100100 11 0010 10 101 ... . .  @rd_pg_rn_e0
 FCVTLT_sd   01100100 11 0010 11 101 ... . .  @rd_pg_rn_e0
diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c
index 40af3024df..46a957b6fb 100644
--- a/target/arm/sve_helper.c
+++ b/target/arm/sve_helper.c
@@ -4708,6 +4708,7 @@ static inline uint64_t vfp_float64_to_uint64_rtz(float64 
f, float_status *s)
 
 DO_ZPZ_FP(sve_fcvt_sh, uint32_t, H1_4, sve_f32_to_f16)
 DO_ZPZ_FP(sve_fcvt_hs, uint32_t, H1_4, sve_f16_to_f32)
+DO_ZPZ_FP(sve_bfcvt,   uint32_t, H1_4, float32_to_bfloat16)
 DO_ZPZ_FP(sve_fcvt_dh, uint64_t, , sve_f64_to_f16)
 DO_ZPZ_FP(sve_fcvt_hd, uint64_t, , sve_f16_to_f64)
 DO_ZPZ_FP(sve_fcvt_ds, uint64_t, , float64_to_float32)
@@ -7740,6 +7741,7 @@ void HELPER(NAME)(void *vd, void *vn, void *vg, void 
*status, uint32_t desc)  \
 } while (i != 0); \
 }
 
+DO_FCVTNT(sve_bfcvtnt,uint32_t, uint16_t, H1_4, H1_2, float32_to_bfloat16)
 DO_FCVTNT(sve2_fcvtnt_sh, uint32_t, uint16_t, H1_4, H1_2, sve_f32_to_f16)
 DO_FCVTNT(sve2_fcvtnt_ds, uint6

[PATCH v2 00/12] target/arm: Implement BFloat16

[Richard Henderson]

Whee!  All prerequisites are now merged.

Patches missing r-b:

  05-softfpu-Add-float_round_to_odd_inf.patch
  06-target-arm-Implement-bfloat16-dot-product-vector.patch
  07-target-arm-Implement-bfloat16-dot-product-indexed.patch
  11-linux-user-aarch64-Enable-hwcap-bits-for-bfloat16.patch
  12-target-arm-Enable-BFloat16-extensions.patch

Per the question of whether additional checks vs VFP or NEON
are required, I have disabled BF16 when either VFP or NEON are
also disabled.  Which seems like a similarly reasonable choice.


r~


Richard Henderson (12):
  target/arm: Add isar_feature_{aa32,aa64,aa64_sve}_bf16
  target/arm: Unify unallocated path in disas_fp_1src
  target/arm: Implement scalar float32 to bfloat16 conversion
  target/arm: Implement vector float32 to bfloat16 conversion
  softfpu: Add float_round_to_odd_inf
  target/arm: Implement bfloat16 dot product (vector)
  target/arm: Implement bfloat16 dot product (indexed)
  target/arm: Implement bfloat16 matrix multiply accumulate
  target/arm: Implement bfloat widening fma (vector)
  target/arm: Implement bfloat widening fma (indexed)
  linux-user/aarch64: Enable hwcap bits for bfloat16
  target/arm: Enable BFloat16 extensions

 include/fpu/softfloat-types.h |   4 +-
 target/arm/cpu.h  |  15 
 target/arm/helper-sve.h   |   4 +
 target/arm/helper.h   |  15 
 target/arm/neon-dp.decode |   1 +
 target/arm/neon-shared.decode |  11 +++
 target/arm/sve.decode |  19 -
 target/arm/vfp.decode |   2 +
 linux-user/elfload.c  |   2 +
 target/arm/cpu.c  |   3 +
 target/arm/cpu64.c|   3 +
 target/arm/cpu_tcg.c  |   1 +
 target/arm/sve_helper.c   |   2 +
 target/arm/translate-a64.c| 142 +-
 target/arm/translate-neon.c   |  91 ++
 target/arm/translate-sve.c| 112 +++
 target/arm/translate-vfp.c|  24 ++
 target/arm/vec_helper.c   | 140 -
 target/arm/vfp_helper.c   |  12 +++
 fpu/softfloat-parts.c.inc |   6 +-
 20 files changed, 584 insertions(+), 25 deletions(-)

-- 
2.25.1

[Bug 1891748] Re: qemu-arm-static 5.1 can't run gcc

[Devaev Maxim]

Okay, what do you think this problem might be related to? I'm glad your
tests are working, but I'm definitely not the only one with this
problem.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1891748

Title:
  qemu-arm-static 5.1 can't run gcc

Status in QEMU:
  Fix Released
Status in Juju Charms Collection:
  New

Bug description:
  Issue discovered while trying to build pikvm (1)

  Long story short: when using qemu-arm-static 5.1, gcc exits whith
  message:

  Allocating guest commpage: Operation not permitted

  
  when using qemu-arm-static v5.0, gcc "works"

  Steps to reproduce will follow

  (1)  https://github.com/pikvm/pikvm/blob/master/pages/building_os.md

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1891748/+subscriptions

Re: [PATCH v6 25/25] python: add tox support

[John Snow]

On 5/25/21 4:46 PM, Cleber Rosa wrote:

On Tue, May 25, 2021 at 04:25:37PM -0400, John Snow wrote:

On 5/25/21 4:15 PM, Cleber Rosa wrote:

On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:

This is intended to be a manually run, non-CI script.

Use tox to test the linters against all python versions from 3.6 to
3.9. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:


sudo dnf install python36 python37 python38 python39


Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.

Signed-off-by: John Snow 
---
   python/README.rst |  2 ++
   python/.gitignore |  1 +
   python/Makefile   |  7 ++-
   python/setup.cfg  |  1 +
   python/tox.ini| 13 +
   5 files changed, 23 insertions(+), 1 deletion(-)
   create mode 100644 python/tox.ini



This works as intended for me.  A couple of notes / suggestions
for future improvements:

   * `dnf install tox` pulled all the Python versions available (I
 assume as suggestions) automatically

   * tox.ini can be folded into setup.cfg



Done!



Nice!


   * a custom container image with all those Python versions may be
 handy for running both the pipenv based job (along with the
 suggestions on the previous patch) and an on-demand,
 "allow_failure" tox based CI job.



Yeah, I was thinking this would be good, too!

I think at this point, it's going to be a follow-up, though. Because
ideally, yes, this SHOULD pass -- it's just that it needs a fairly
particular environment to run in, which is annoying, so it's here as an
optional-ish thing for now.

Maybe I'll make a new fedora:latest container that's meant solely for
testing python stuff, because it's just such a convenient distro for it.

Later, though.



Sure.


Other than those suggestions, this LGTM!

Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 



🎉


\o/ (with 3 characters, because I'm unable to find the right codepoint)



if you're on fedora using the ime-bus stuff, try ctrl+shift+e to enter 
EMOJI MODE and then type 'tada' and then hit 'space' to search.


🥳

Re: [PATCH] target/riscv: Pass the same value to oprsz and maxsz.

[Alistair Francis]

On Fri, May 21, 2021 at 3:48 PM LIU Zhiwei  wrote:
>
> Since commit e2e7168a214b0ed98dc357bba96816486a289762, if oprsz
> is still zero(as we don't use this field), simd_desc will trigger an
> assert.
>
> Besides, tcg_gen_gvec_*_ptr calls simd_desc in it's implementation.
> Here we pass the value to maxsz and oprsz to bypass the assert.
>
> Signed-off-by: LIU Zhiwei 

Thanks!

Applied to riscv-to-apply.next

Alistair

> ---
>  target/riscv/insn_trans/trans_rvv.c.inc | 89 ++---
>  1 file changed, 50 insertions(+), 39 deletions(-)
>
> diff --git a/target/riscv/insn_trans/trans_rvv.c.inc 
> b/target/riscv/insn_trans/trans_rvv.c.inc
> index 47914a3b69..83d9a285ba 100644
> --- a/target/riscv/insn_trans/trans_rvv.c.inc
> +++ b/target/riscv/insn_trans/trans_rvv.c.inc
> @@ -183,7 +183,7 @@ static bool ldst_us_trans(uint32_t vd, uint32_t rs1, 
> uint32_t data,
>   * The first part is vlen in bytes, encoded in maxsz of simd_desc.
>   * The second part is lmul, encoded in data of simd_desc.
>   */
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  gen_get_gpr(base, rs1);
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
> @@ -334,7 +334,7 @@ static bool ldst_stride_trans(uint32_t vd, uint32_t rs1, 
> uint32_t rs2,
>  mask = tcg_temp_new_ptr();
>  base = tcg_temp_new();
>  stride = tcg_temp_new();
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  gen_get_gpr(base, rs1);
>  gen_get_gpr(stride, rs2);
> @@ -462,7 +462,7 @@ static bool ldst_index_trans(uint32_t vd, uint32_t rs1, 
> uint32_t vs2,
>  mask = tcg_temp_new_ptr();
>  index = tcg_temp_new_ptr();
>  base = tcg_temp_new();
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  gen_get_gpr(base, rs1);
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
> @@ -594,7 +594,7 @@ static bool ldff_trans(uint32_t vd, uint32_t rs1, 
> uint32_t data,
>  dest = tcg_temp_new_ptr();
>  mask = tcg_temp_new_ptr();
>  base = tcg_temp_new();
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  gen_get_gpr(base, rs1);
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
> @@ -671,7 +671,7 @@ static bool amo_trans(uint32_t vd, uint32_t rs1, uint32_t 
> vs2,
>  mask = tcg_temp_new_ptr();
>  index = tcg_temp_new_ptr();
>  base = tcg_temp_new();
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  gen_get_gpr(base, rs1);
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
> @@ -831,7 +831,7 @@ do_opivv_gvec(DisasContext *s, arg_rmrr *a, GVecGen3Fn 
> *gvec_fn,
>  data = FIELD_DP32(data, VDATA, LMUL, s->lmul);
>  tcg_gen_gvec_4_ptr(vreg_ofs(s, a->rd), vreg_ofs(s, 0),
> vreg_ofs(s, a->rs1), vreg_ofs(s, a->rs2),
> -   cpu_env, 0, s->vlen / 8, data, fn);
> +   cpu_env, s->vlen / 8, s->vlen / 8, data, fn);
>  }
>  gen_set_label(over);
>  return true;
> @@ -874,7 +874,7 @@ static bool opivx_trans(uint32_t vd, uint32_t rs1, 
> uint32_t vs2, uint32_t vm,
>  data = FIELD_DP32(data, VDATA, MLEN, s->mlen);
>  data = FIELD_DP32(data, VDATA, VM, vm);
>  data = FIELD_DP32(data, VDATA, LMUL, s->lmul);
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
>  tcg_gen_addi_ptr(src2, cpu_env, vreg_ofs(s, vs2));
> @@ -1021,7 +1021,7 @@ static bool opivi_trans(uint32_t vd, uint32_t imm, 
> uint32_t vs2, uint32_t vm,
>  data = FIELD_DP32(data, VDATA, MLEN, s->mlen);
>  data = FIELD_DP32(data, VDATA, VM, vm);
>  data = FIELD_DP32(data, VDATA, LMUL, s->lmul);
> -desc = tcg_const_i32(simd_desc(0, s->vlen / 8, data));
> +desc = tcg_const_i32(simd_desc(s->vlen / 8, s->vlen / 8, data));
>
>  tcg_gen_addi_ptr(dest, cpu_env, vreg_ofs(s, vd));
>  tcg_gen_addi_ptr(src2, cpu_env, vreg_ofs(s, vs2));
> @@ -1119,7 +1119,7 @@ static bool do_opivv_widen(DisasContext *s, arg_rmrr *a,
>  tcg_gen_gvec_4_ptr(vreg_ofs(s, a->rd), vreg_ofs(s, 0),
> vreg_ofs(s, a->rs1),
> vreg_ofs(s, a->rs2),
> -   cpu_env, 0, s->vlen / 8,
> +   cpu_env, s->vlen / 8, s->vlen / 8,
> data, fn);
>  gen_set_label(over);
>  return true;
> @@ -1207,7 +1207,7 @@ static bool do_opiwv_widen(DisasContext *s, arg_rmrr *a,
>  tcg_gen_gvec_4_ptr(vreg_ofs(s, a->rd), vreg_of

Re: [PATCH v1 1/1] target/riscv/pmp: Add assert for ePMP operations

[Alistair Francis]

On Fri, May 21, 2021 at 8:56 AM Alistair Francis
 wrote:
>
> Although we construct epmp_operation in such a way that it can only be
> between 0 and 15 Coverity complains that we don't handle the other
> possible cases. To fix Coverity and make it easier for humans to read
> add a default case to the switch statement that calls
> g_assert_not_reached().
>
> Fixes: CID 1453108
> Signed-off-by: Alistair Francis 

Thanks!

Applied to riscv-to-apply.next

Alistair

> ---
>  target/riscv/pmp.c | 4 
>  1 file changed, 4 insertions(+)
>
> diff --git a/target/riscv/pmp.c b/target/riscv/pmp.c
> index 78203291de..82ed020b10 100644
> --- a/target/riscv/pmp.c
> +++ b/target/riscv/pmp.c
> @@ -402,6 +402,8 @@ bool pmp_hart_has_privs(CPURISCVState *env, target_ulong 
> addr,
>  case 15:
>  *allowed_privs = PMP_READ;
>  break;
> +default:
> +g_assert_not_reached();
>  }
>  } else {
>  switch (epmp_operation) {
> @@ -433,6 +435,8 @@ bool pmp_hart_has_privs(CPURISCVState *env, target_ulong 
> addr,
>  case 7:
>  *allowed_privs = PMP_READ | PMP_WRITE | PMP_EXEC;
>  break;
> +default:
> +g_assert_not_reached();
>  }
>  }
>  }
> --
> 2.31.1
>

Re: [PULL v3 36/42] target/riscv: Remove the hardcoded MSTATUS_SD macro

[Alistair Francis]

On Fri, May 21, 2021 at 12:09 PM LIU Zhiwei  wrote:
>
>
> On 5/21/21 6:55 AM, Alistair Francis wrote:
> > On Thu, May 20, 2021 at 11:55 PM Peter Maydell  
> > wrote:
> >> On Tue, 11 May 2021 at 11:22, Alistair Francis  
> >> wrote:
> >>> Signed-off-by: Alistair Francis 
> >>> Reviewed-by: Richard Henderson 
> >>> Message-id: 
> >>> fcc125d96da941b56c817c9dd6068dc36478fc53.1619234854.git.alistair.fran...@wdc.com
> >>> ---
> >>>   target/riscv/cpu_bits.h  | 10 --
> >>>   target/riscv/csr.c   | 12 ++--
> >>>   target/riscv/translate.c | 19 +--
> >>>   3 files changed, 27 insertions(+), 14 deletions(-)
> >>> diff --git a/target/riscv/translate.c b/target/riscv/translate.c
> >>> index 26eccc5eb1..a596f80f20 100644
> >>> --- a/target/riscv/translate.c
> >>> +++ b/target/riscv/translate.c
> >>> @@ -78,6 +78,17 @@ static inline bool has_ext(DisasContext *ctx, uint32_t 
> >>> ext)
> >>>   return ctx->misa & ext;
> >>>   }
> >>>
> >>> +#ifdef TARGET_RISCV32
> >>> +# define is_32bit(ctx)  true
> >>> +#elif defined(CONFIG_USER_ONLY)
> >>> +# define is_32bit(ctx)  false
> >>> +#else
> >>> +static inline bool is_32bit(DisasContext *ctx)
> >>> +{
> >>> +return (ctx->misa & RV32) == RV32;
> >>> +}
> >>> +#endif
> >> Hi; Coverity points out (CID 1453107) that this is_32bit() function
> >> can never return true for at least some build configs, because RV32
> >> is defined as ((target_ulong)1 << (TARGET_LONG_BITS - 2))
> >> but ctx->misa is a uint32_t field, which (if TARGET_LONG_BITS is
> >> 64) is not big enough for the RV32 bit.
> > This seems like a false positive as RV32 is defined as:
> >
> > ((target_ulong)1 << (TARGET_LONG_BITS - 2))
> >
> > while ctx->misa is a target_ulong.
>
> Although the misa in RISCVCPUState is target_ulong, the misa in
> DisasContext is uint32_t.
>
> I think we should  fix up the misa in DisasContext.

Good catch, I'll send a patch.

Alistair

>
> Zhiwei
>
> >
> > So it should always be able to return true.
> >
> > Alistair
> >
> >> Bug, or false positive ?
> >>
> >> thanks
> >> -- PMM

Re: [PATCH v2 3/7] ACPI ERST: support for ACPI ERST feature

[Igor Mammedov]

On Tue, 25 May 2021 20:22:26 +
Eric DeVolder  wrote:

> Igor,
> Thank you for the pointers, I've turned the corner on the use of hostmem-file 
> and am able to use it now!
> I do have one question regarding hostmem-file. In the VMStateDescription, I 
> used to have this for the contents
> of the nvram (backed by a file):
> 
>   VMSTATE_VARRAY_UINT32(nvram, ERSTDeviceState, prop_size, 0,
> vmstate_info_uint8, uint8_t),
> 
> I've searched and I do not locate an example of passing a HostMemoryBackend 
> object; how do I do that?

if I'm not wrong, you do not need to do it, i.e. treat backing file as any 
other storage,
i.e. put it on shared storage were VM's hard disks are located so source and 
target host can access it.

alternatively you can mark MemoryRegion as migratable, which will make QEMU 
stream its content
over migration socket automatically. In that case file associated with backend 
doesn't need to
be shared between source and target hosts. I think this option is fine if file 
in question is small,
but others might think otherwise. (obvious drawback is that it increases 
migration time)

CCed David as he might have an opinion about it from migration point of view.


> Thanks,
> eric
> 
> 
[...]

Re: [PATCH v6 25/25] python: add tox support

[Cleber Rosa]

On Tue, May 25, 2021 at 04:25:37PM -0400, John Snow wrote:
> On 5/25/21 4:15 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
> > > This is intended to be a manually run, non-CI script.
> > > 
> > > Use tox to test the linters against all python versions from 3.6 to
> > > 3.9. This will only work if you actually have those versions installed
> > > locally, but Fedora makes this easy:
> > > 
> > > > sudo dnf install python36 python37 python38 python39
> > > 
> > > Unlike the pipenv tests (make venv-check), this pulls "whichever"
> > > versions of the python packages, so they are unpinned and may break as
> > > time goes on. In the case that breakages are found, setup.cfg should be
> > > amended accordingly to avoid the bad dependant versions, or the code
> > > should be amended to work around the issue.
> > > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/README.rst |  2 ++
> > >   python/.gitignore |  1 +
> > >   python/Makefile   |  7 ++-
> > >   python/setup.cfg  |  1 +
> > >   python/tox.ini| 13 +
> > >   5 files changed, 23 insertions(+), 1 deletion(-)
> > >   create mode 100644 python/tox.ini
> > > 
> > 
> > This works as intended for me.  A couple of notes / suggestions
> > for future improvements:
> > 
> >   * `dnf install tox` pulled all the Python versions available (I
> > assume as suggestions) automatically
> > 
> >   * tox.ini can be folded into setup.cfg
> > 
> 
> Done!
>

Nice!

> >   * a custom container image with all those Python versions may be
> > handy for running both the pipenv based job (along with the
> > suggestions on the previous patch) and an on-demand,
> > "allow_failure" tox based CI job.
> > 
> 
> Yeah, I was thinking this would be good, too!
> 
> I think at this point, it's going to be a follow-up, though. Because
> ideally, yes, this SHOULD pass -- it's just that it needs a fairly
> particular environment to run in, which is annoying, so it's here as an
> optional-ish thing for now.
> 
> Maybe I'll make a new fedora:latest container that's meant solely for
> testing python stuff, because it's just such a convenient distro for it.
> 
> Later, though.
>

Sure.

> > Other than those suggestions, this LGTM!
> > 
> > Reviewed-by: Cleber Rosa 
> > Tested-by: Cleber Rosa 
> > 
> 
> 🎉

\o/ (with 3 characters, because I'm unable to find the right codepoint)


signature.asc
Description: PGP signature

Re: [PATCH v6 22/25] python: add Makefile for some common tasks

[Cleber Rosa]

On Tue, May 25, 2021 at 03:45:26PM -0400, John Snow wrote:
> On 5/25/21 3:24 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:38PM -0400, John Snow wrote:
> > > Add "make venv" to create the pipenv-managed virtual environment that
> > > contains our explicitly pinned dependencies.
> > > 
> > > Add "make check" to run the python linters [in the host execution
> > > environment].
> > > 
> > > Add "make venv-check" which combines the above two: create/update the
> > > venv, then run the linters in that explicitly managed environment.
> > > 
> > > Add "make develop" which canonizes the runes needed to get both the
> > > linting pre-requisites (the "[devel]" part), and the editable
> > > live-install (the "-e" part) of these python libraries.
> > > 
> > > make clean: delete miscellaneous python packaging output possibly
> > > created by pipenv, pip, or other python packaging utilities
> > > 
> > > make distclean: delete the above, the .venv, and the editable "qemu"
> > > package forwarder (qemu.egg-info) if there is one.
> > > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/README.rst |  3 +++
> > >   python/Makefile   | 42 ++
> > >   2 files changed, 45 insertions(+)
> > >   create mode 100644 python/Makefile
> > > 
> > > diff --git a/python/README.rst b/python/README.rst
> > > index e107bd12a69..3e09d20c23c 100644
> > > --- a/python/README.rst
> > > +++ b/python/README.rst
> > > @@ -35,6 +35,9 @@ Files in this directory
> > >   - ``qemu/`` Python package source directory.
> > >   - ``tests/`` Python package tests directory.
> > >   - ``avocado.cfg`` Configuration for the Avocado test-runner.
> > > +  Used by ``make check`` et al.
> > > +- ``Makefile`` provides some common testing/installation invocations.
> > > +  Try ``make help`` to see available targets.
> > >   - ``MANIFEST.in`` is read by python setuptools, it specifies additional 
> > > files
> > > that should be included by a source distribution.
> > >   - ``PACKAGE.rst`` is used as the README file that is visible on 
> > > PyPI.org.
> > > diff --git a/python/Makefile b/python/Makefile
> > > new file mode 100644
> > > index 000..184f59e5634
> > > --- /dev/null
> > > +++ b/python/Makefile
> > > @@ -0,0 +1,42 @@
> > > +.PHONY: help venv venv-check check clean distclean develop
> > > +
> > > +help:
> > > + @echo "python packaging help:"
> > > + @echo ""
> > > + @echo "make venv:   Create pipenv's virtual environment."
> > > + @echo "NOTE: Requires Python 3.6 and pipenv."
> > > + @echo "  Will download packages from PyPI."
> > > + @echo "Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
> > > + @echo ""
> > > + @echo "make venv-check: run linters using pipenv's virtual environment."
> > > + @echo "Hint: If you don't know which test to run, run this one!"
> > > + @echo ""
> > > + @echo "make develop:Install deps for 'make check', and"
> > > + @echo " the qemu libs in editable/development mode."
> > > + @echo ""
> > > + @echo "make check:  run linters using the current environment."
> > > + @echo ""
> > 
> > Let's observe how this will be used (or misused).  I fear most people
> > will jump into `make check`, even though you have described `make
> > venv-check` as the primary choice.
> > 
> > We have a precedent with `make check-acceptance` that will create a
> > venv and use it by default, so we can consider that as a fallback
> > strategy based on user feedback.
> > 
> 
> Right, I see. Though, I did intentionally want to make it clear which of
> these invocations created an environment and which did not.
> 
> Unlike the acceptance tests, it might make sense to run these tests both
> inside and outside of that venv, so I opted to make the default "make"
> target "make help".
> 
> The Gitlab CI will run the right one, after all -- and I do still expect the
> regular 'make check' to pass, so I am not as sure that it's a crucial
> failure if someone runs the "wrong one".
> 
> > > + @echo "make clean:  remove build output."
> > > + @echo ""
> > > + @echo "make distclean:  remove venv files, qemu package forwarder, and"
> > > + @echo " everything from 'make clean'."
> > > +
> > > +venv: .venv
> > > +.venv: Pipfile.lock
> > > + @PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
> > > + @touch .venv
> > > +
> > > +venv-check: venv
> > > + @pipenv run make check
> > > +
> > > +develop:
> > > + pip3 install -e .[devel]
> > > +
> > > +check:
> > > + @avocado --config avocado.cfg run tests/
> > > +
> > > +clean:
> > > + rm -rf build/ dist/
> > > +
> > 
> > Usually `python3 setup.py clean --all` would be the better choice here,
> > but, it doesn't clean `dist/`, so I'm OK with this.
> > 
> 
> Hm, I should probably move the 'dist' down into 'distclean' anyway, and I
> will replace the clean invocation with the one you suggest.
>

OK then, makes sense to me.

- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH v6 23/25] python: add .gitignore

[Cleber Rosa]

On Tue, May 25, 2021 at 04:10:55PM -0400, John Snow wrote:
> On 5/25/21 3:36 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:
> > > Ignore *Python* build and package output (build, dist, qemu.egg-info);
> > > these files are not created as part of a QEMU build.
> > > 
> > > Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
> > > .mypy_cache).
> > > 
> > > Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
> > > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/.gitignore | 19 +++
> > >   1 file changed, 19 insertions(+)
> > >   create mode 100644 python/.gitignore
> > > 
> > > diff --git a/python/.gitignore b/python/.gitignore
> > > new file mode 100644
> > > index 000..e27c99e009c
> > > --- /dev/null
> > > +++ b/python/.gitignore
> > > @@ -0,0 +1,19 @@
> > > +# python bytecode cache
> > > +*.pyc
> > 
> > This is a duplicate from the parent .gitignore, so I would avoid it.
> > 
> > > +__pycache__/
> > 
> > And this one is interesting because, the only thing that *should* be
> > in __pycache__ dirs is .pyc files (covered by the parent .gitignore
> > file).
> > 
> > So, I get the same behavior without these two entries here, so I would
> > skip them.  Let me know if you have any reason for explicitly
> > including them.
> > 
> > - Cleber.
> > 
> 
> Hm, not really ... Just completeness, I suppose, since this directory is
> becoming increasingly separate from the rest of the tree.
> 
> It isn't crucial, it just seemed like a weird omission if they weren't
> listed here. *shrug*
> 
> --js

And still, this dir is part of the overall tree.  Honestly, without
any change in behavior, I'd *not* add those two ignore rules.

Cheers,
- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH v6 20/25] python: add devel package requirements to setuptools

[Cleber Rosa]

On Tue, May 25, 2021 at 01:43:42PM -0400, John Snow wrote:
> On 5/25/21 12:13 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:36PM -0400, John Snow wrote:
> > > setuptools doesn't have a formal understanding of development requires,
> > > but it has an optional feataures section. Fine; add a "devel" feature
> > > and add the requirements to it.
> > > 
> > > To avoid duplication, we can modify pipenv to install qemu[devel]
> > > instead. This enables us to run invocations like "pip install -e
> > > .[devel]" and test the package on bleeding-edge packages beyond those
> > > specified in Pipfile.lock.
> > > 
> > > Importantly, this also allows us to install the qemu development
> > > packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
> > > will now fail if the proper development dependencies are not already
> > > met. This can be useful for automated build scripts where fetching
> > > network packages may be undesirable.
> > > 
> > 
> > This is a fairly exotic feature of setuptools, with very very few
> > packages that I know about using it.  With most users (I believe)
> > relying on pipenv to get the exact packages, the setuptools/pip use
> > case may fall into obscurity IMO.
> > 
> 
> Fair enough.
> 
> The intent is:
> 
> - Pipenv is more for CI, to deploy a consistent set of frozen packages that
> are known to behave in an extremely stable manner. My hope is to avoid
> breaking changes introduced unknowingly by pylint et al.
> 
> - pip install qemu[devel] is intended more for external/normal use by
> developers. It grabs the latest and greatest and it may indeed break as
> dependencies change beyond my awareness.
> 
> 
> Some packages like aiohttp use that optional dependency feature to install
> optional modules -- `pip install aiohttp[speedups]` installs optional
> dependencies that allow that module to work much faster, but aren't
> required.
> 
> Since these linting tools aren't *required* just to *use* the package, I am
> doing users a courtesy by listing them as optional. That way, they aren't
> pulled in when using "pip install qemu", and if I have to pin on specific
> sub-versions etc, it won't include conflict dependencies for people using
> other projects that DO declare a hard requirement on those packages.
> 
> I can amend the PACKAGE.rst file to mention this usage, though it's only
> useful for folks developing the package.
> 
> (Still, part of the ploy here is to attract outside help on developing the
> QEMU SDK, pull requests welcome etc, so it's worth a documentation blurb for
> now.)
>

Yep, I agree with your reasoning here.  I just felt like an extra bit
of documentation would do the trick.

> > So my suggestion is: consider better exposing the fact that this is
> > available (a documentation section perhaps).
> > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/Pipfile  |  5 +
> > >   python/Pipfile.lock | 14 +-
> > >   python/setup.cfg|  9 +
> > >   3 files changed, 19 insertions(+), 9 deletions(-)
> > > 
> > 
> > Either way,
> > 
> > Reviewed-by: Cleber Rosa 
> > 
> 
> Thanks! I am taking your R-B and I have applied the following diff.
> 
> Note that the PACKAGE.rst blurb references qemu[devel] instead because the
> PACKAGE.rst file is what is displayed theoretically on PyPI. That exact
> invocation will fail currently, because it's not on PyPI yet.
> 
> A little weird, but I *think* it's correct.
> 
> 
> diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
> index 1bbfe1b58e2..05ea7789fc1 100644
> --- a/python/PACKAGE.rst
> +++ b/python/PACKAGE.rst
> @@ -31,3 +31,7 @@ official `GitLab mirror
> `_.
>  Please report bugs on the `QEMU issue tracker
>  `_ and tag ``@jsnow`` in
>  the report.
> +
> +Optional packages necessary for running code quality analysis for this
> +package can be installed with the optional dependency group "devel":
> +``pip install qemu[devel]``.
> diff --git a/python/README.rst b/python/README.rst
> index bf9bbca979a..954870973d0 100644
> --- a/python/README.rst
> +++ b/python/README.rst
> @@ -24,6 +24,10 @@ which installs a version of the package that installs a
> forwarder
>  pointing to these files, such that the package always reflects the
>  latest version in your git tree.
> 
> +Installing ".[devel]" instead of "." will additionally pull in required
> +packages for testing this package. They are not runtime requirements,
> +and are not needed to simply use these libraries.
> +
>  See `Installing packages using pip and virtual environments
> 
> `_
>  for more information.

Looks great to me.  And, let me be clear about it:

Reviewed-by: Cleber Rosa 


signature.asc
Description: PGP signature

Re: [PATCH v6 18/25] python/qemu: add isort to pipenv

[Cleber Rosa]

On Tue, May 25, 2021 at 01:21:25PM -0400, John Snow wrote:
> On 5/25/21 11:56 AM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:34PM -0400, John Snow wrote:
> > > isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
> > > certain "from ..." clauses that are not related to imports.
> > > 
> > > isort < 5.1.1 has a bug where it does not handle comments near import
> > > statements correctly.
> > > 
> > > Require 5.1.2 or greater.
> > > 
> > > isort can be run with 'isort -c qemu' from the python root.
> > > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/Pipfile  | 1 +
> > >   python/Pipfile.lock | 4 ++--
> > >   2 files changed, 3 insertions(+), 2 deletions(-)
> > 
> > Reviewed-by: Cleber Rosa 
> > 
> 
> Thanks. I have also updated the commit message a little bit:
> 
> isort can be run (in "check" mode) with 'isort -c qemu' from the python
> root. isort can also be used to fix/rewrite import order automatically
> by using 'isort qemu'.
> 
> --js

OK, sounds even better!

- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH v6 24/25] gitlab: add python linters to CI

[John Snow]

On 5/25/21 3:55 PM, Cleber Rosa wrote:

On Wed, May 12, 2021 at 07:12:40PM -0400, John Snow wrote:

Add python3.6 to the fedora container image: we need it to run the
linters against that explicit version to make sure we don't break our
minimum version promise.

Add pipenv so that we can fetch precise versions of pip packages we need
to guarantee test reproducability.

Signed-off-by: John Snow 
---
  .gitlab-ci.yml | 12 
  tests/docker/dockerfiles/fedora.docker |  2 ++
  2 files changed, 14 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index dcb6317aace..a371c0c7163 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -779,6 +779,18 @@ check-patch:
  GIT_DEPTH: 1000
allow_failure: true
  
+

+check-python:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  script:
+- cd python
+- make venv-check
+  variables:
+GIT_DEPTH: 1000
+  needs:
+job: amd64-fedora-container
+
  check-dco:
stage: build
image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
diff --git a/tests/docker/dockerfiles/fedora.docker 
b/tests/docker/dockerfiles/fedora.docker
index 915fdc1845e..6908d69ac37 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -84,6 +84,7 @@ ENV PACKAGES \
  numactl-devel \
  perl \
  perl-Test-Harness \
+pipenv \
  pixman-devel \
  python3 \
  python3-PyYAML \
@@ -93,6 +94,7 @@ ENV PACKAGES \
  python3-pip \
  python3-sphinx \
  python3-virtualenv \
+python3.6 \


I personally would prefer having a different container image for this
job.  Because it would:

1. Be super simple (FROM fedora:33 / dnf -y install python3.6 pipenv)
2. Not carry all this unnecessary baggage
3. Not risk building QEMU with Python 3.6 (suppose the ./configure
probe changes unintentionally)

But, AFAICT there is no precedent in requiring new images for
different types of checks.  So, unless someone else complains loudly,
I'm OK with this.

Reviewed-by: Cleber Rosa 



Hm

I'll try. You're right that it would make the pipeline a lot simpler 
because it'd have less cruft.


but thank you for the R-B in the meantime ;)

--js

Re: [PATCH v6 06/25] python: add directory structure README.rst files

[Cleber Rosa]

On Tue, May 25, 2021 at 01:14:50PM -0400, John Snow wrote:
> On 5/24/21 10:33 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:22PM -0400, John Snow wrote:
> > > Add short readmes to python/, python/qemu/, python/qemu/machine,
> > > python/qemu/qmp, and python/qemu/utils that explain the directory
> > > hierarchy. These readmes are visible when browsing the source on
> > > e.g. gitlab/github and are designed to help new developers/users quickly
> > > make sense of the source tree.
> > > 
> > > They are not designed for inclusion in a published manual.
> > > 
> > > Signed-off-by: John Snow 
> > > ---
> > >   python/README.rst  | 41 ++
> > >   python/qemu/README.rst |  8 +++
> > >   python/qemu/machine/README.rst |  9 
> > >   python/qemu/qmp/README.rst |  9 
> > >   python/qemu/utils/README.rst   |  7 ++
> > >   5 files changed, 74 insertions(+)
> > >   create mode 100644 python/README.rst
> > >   create mode 100644 python/qemu/README.rst
> > >   create mode 100644 python/qemu/machine/README.rst
> > >   create mode 100644 python/qemu/qmp/README.rst
> > >   create mode 100644 python/qemu/utils/README.rst
> > > 
> > > diff --git a/python/README.rst b/python/README.rst
> > > new file mode 100644
> > > index 000..7a0dc5dff4a
> > > --- /dev/null
> > > +++ b/python/README.rst
> > > @@ -0,0 +1,41 @@
> > > +QEMU Python Tooling
> > > +===
> > > +
> > > +This directory houses Python tooling used by the QEMU project to build,
> > > +configure, and test QEMU. It is organized by namespace (``qemu``), and
> > > +then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
> > > +
> > > +``setup.py`` is used by ``pip`` to install this tooling to the current
> > > +environment. ``setup.cfg`` provides the packaging configuration used by
> > > +setup.py in a setuptools specific format. You will generally invoke it
> > 
> > For consistency, ``setup.py`` here?  Also, I guess ``setuptools`` as it
> > falls in the same category of ``pip``.
> > 
> 
> Kinda-sorta, but `pip` is a command line executable and setuptools isn't.
> Along those lines I'll fix setup.py but leave setuptools as-is.
>

OK, fair enough.

> > > +by doing one of the following:
> > > +
> > > +1. ``pip3 install .`` will install these packages to your current
> > > +   environment. If you are inside a virtual environment, they will
> > > +   install there. If you are not, it will attempt to install to the
> > > +   global environment, which is not recommended.
> > 
> > Maybe some **emphasis** on **not**?
> > 
> 
> Sure :)
> 
> > > +
> > > +2. ``pip3 install --user .`` will install these packages to your user's
> > > +   local python packages. If you are inside of a virtual environment,
> > > +   this will fail.
> > > +
> > 
> > Maybe note that, if you are inside of a virtual environment, option #1
> > will probably be what users doing "--user" in a venv actually want.
> > 
> 
> Yes. It's frequently annoying how this works, because it's hard to relay
> succinctly :)
> 
> I think at least newer versions of pip give you good warnings when you use
> --user for virtual environments at least.
>

True.

> > > +If you append the ``-e`` argument, pip will install in "editable" mode;
> > > +which installs a version of the package that installs a forwarder
> > > +pointing to these files, such that the package always reflects the
> > > +latest version in your git tree.
> > > +
> > > +See `Installing packages using pip and virtual environments
> > > +`_
> > > +for more information.
> > > +
> > > +
> > > +Files in this directory
> > > +---
> > > +
> > > +- ``qemu/`` Python package source directory.
> > > +- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
> > > +- ``README.rst`` you are here!
> > > +- ``VERSION`` contains the PEP-440 compliant version used to describe
> > > +  this package; it is referenced by ``setup.cfg``.
> > > +- ``setup.cfg`` houses setuptools package configuration.
> > > +- ``setup.py`` is the setuptools installer used by pip; See above.
> > 
> > Not only used by pip... but I understand the reason for limiting the
> > amount of information given here.
> > 
> 
> Yes ... suggesting broadly that I don't really support using
> setuptools/setup.py alone to install the package, but instead expect and
> consider 'pip' to be the canonical/supported interface.
> 
> There are sometimes minor differences between how they handle things, so I
> wanted less emphasis on setuptools et al.
>

Sure, I can definitely side with you here.

> > > diff --git a/python/qemu/README.rst b/python/qemu/README.rst
> > > new file mode 100644
> > > index 000..d04943f526c
> > > --- /dev/null
> > > +++ b/python/qemu/README.rst
> > > @@ -0,0 +1,8 @@
> > > +QEMU Python Namespace
> > > +=
> > > +
> > > +This directory serves as th

Re: [PATCH v6 25/25] python: add tox support

[John Snow]

On 5/25/21 4:15 PM, Cleber Rosa wrote:

On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:

This is intended to be a manually run, non-CI script.

Use tox to test the linters against all python versions from 3.6 to
3.9. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:


sudo dnf install python36 python37 python38 python39


Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.

Signed-off-by: John Snow 
---
  python/README.rst |  2 ++
  python/.gitignore |  1 +
  python/Makefile   |  7 ++-
  python/setup.cfg  |  1 +
  python/tox.ini| 13 +
  5 files changed, 23 insertions(+), 1 deletion(-)
  create mode 100644 python/tox.ini



This works as intended for me.  A couple of notes / suggestions
for future improvements:

  * `dnf install tox` pulled all the Python versions available (I
assume as suggestions) automatically

  * tox.ini can be folded into setup.cfg



Done!


  * a custom container image with all those Python versions may be
handy for running both the pipenv based job (along with the
suggestions on the previous patch) and an on-demand,
"allow_failure" tox based CI job.



Yeah, I was thinking this would be good, too!

I think at this point, it's going to be a follow-up, though. Because 
ideally, yes, this SHOULD pass -- it's just that it needs a fairly 
particular environment to run in, which is annoying, so it's here as an 
optional-ish thing for now.


Maybe I'll make a new fedora:latest container that's meant solely for 
testing python stuff, because it's just such a convenient distro for it.


Later, though.


Other than those suggestions, this LGTM!

Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 



🎉

Re: [PATCH v2 3/7] ACPI ERST: support for ACPI ERST feature

[Eric DeVolder]

Igor,
Thank you for the pointers, I've turned the corner on the use of hostmem-file 
and am able to use it now!
I do have one question regarding hostmem-file. In the VMStateDescription, I 
used to have this for the contents
of the nvram (backed by a file):

  VMSTATE_VARRAY_UINT32(nvram, ERSTDeviceState, prop_size, 0,
vmstate_info_uint8, uint8_t),

I've searched and I do not locate an example of passing a HostMemoryBackend 
object; how do I do that?
Thanks,
eric




From: Igor Mammedov 
Sent: Thursday, May 20, 2021 6:00 AM
To: Eric DeVolder 
Cc: ehabk...@redhat.com ; m...@redhat.com 
; Konrad Wilk ; qemu-devel@nongnu.org 
; pbonz...@redhat.com ; Boris 
Ostrovsky ; r...@twiddle.net ; 
jus...@redhat.com 
Subject: Re: [PATCH v2 3/7] ACPI ERST: support for ACPI ERST feature

On Tue, 18 May 2021 17:08:31 +
Eric DeVolder  wrote:

> Hi Igor,
> Thanks for the information. I am primarily interested in ensuring data 
> persistence in the case of #1.
> As it stands so far, I have yet to observe any kind of write back into the 
> backing file. Just to summarize,
> what I've done thus far is:
>
> in erst_realizefn():
> ...
> s->hostmem_obj = object_new(TYPE_MEMORY_BACKEND_FILE);
> object_property_set_str(s->hostmem_obj, "mem-path", (const char 
> *)(TYPE_ACPI_ERST ".hostmem"), &error_fatal);
> object_property_set_int(s->hostmem_obj, "size", s->prop_size, 
> &error_fatal);
> user_creatable_complete(USER_CREATABLE(s->hostmem_obj), &error_fatal);
> s->hostmem = MEMORY_BACKEND(s->hostmem_obj);

backend should be provided by user on CLI so all backend's properties are 
configured there
as user desires and frontend should access it via link property.
see how pc-dimm's memdev property is used.

> and then in erst_update_backing_file(), which is called when records are 
> created/updated:
>
> ...
> if ((mr = host_memory_backend_get_memory(s->hostmem))) {
> uint8_t *p = (uint8_t *)memory_region_get_ram_ptr(mr);
> memcpy(p + offset, data, length);
> memory_region_msync(mr, 0, s->prop_size); /* for now, the whole thing 
> */
> }
>
> I've instrumented this code, and I can see the records. I've instrumented 
> memory_region_msync() all the way down
> to qemu_msync() and it makes it into that code. But the end result has always 
> been the same, the backing file is
> never updated.
>
> I'm not really sure what else I need to do to get the hostmem contents to be 
> written back into the file.

see "man mmap"
 in particular MAP_SHARED vs MAP_PRIVATE

and there is a corresponding property for the file backend to manage that.

in case #1 no explicit sync is needed, backing file should be updated on close 
at the latest
(whether it's graceful/or forced (i.e. crash))


> Thanks,
> eric
>
>
> 
> From: Igor Mammedov 
> Sent: Monday, May 17, 2021 11:31 AM
> To: Eric DeVolder 
> Cc: ehabk...@redhat.com ; m...@redhat.com 
> ; Konrad Wilk ; 
> qemu-devel@nongnu.org ; pbonz...@redhat.com 
> ; Boris Ostrovsky ; 
> r...@twiddle.net ; jus...@redhat.com 
> Subject: Re: [PATCH v2 3/7] ACPI ERST: support for ACPI ERST feature
>
> On Mon, 17 May 2021 15:01:02 +
> Eric DeVolder  wrote:
>
> > Hi Igor,
> > I've been working to transition ERST to use the hostmem-file object as the 
> > backing store, as requested.
> >
> > I have the backend-file object now in ERST, and I have a question for you. 
> > This hostmem-file initializes
> > itself from a file, but in looking at the code, I do not see that it ever 
> > writes back to the file!? Furthermore,
> > I don't see a "flush" type method to force writeback of data in the object 
> > back to file?
> >
> > The original ERST code would flush/write to the backing file each record as 
> > it was created. I don't see
> > any equivalent way of doing that with hostmem-file?
>
> To force flush you can use memory_region_msync() on MemoryRegion that you get 
> from hostmem backend.
> But question is what are you trying to achieve with sync
>   1. data persistence in case of QEMU crash
>   2. data persistence in case of host crash
>
> for the former you do not need explicit sync as memory buffers should be 
> flushed to disk by kernel
> if you put backend on nvdimm, you should get 2 without sync as well (see 
> pmem=on property)
>
> just do not forget that sync is not free, so if #1 is acceptable I'd avoid 
> explicit sync.
>
>
> > Please point out where I am misunderstanding.
> >
> > Thanks,
> > eric
[...]

Re: [PATCH 5/5] linux-user: Implement pivot_root

[Laurent Vivier]

Le 24/05/2021 à 06:54, YAMAMOTO Takashi a écrit :
> Used by runc.
> 
> Signed-off-by: YAMAMOTO Takashi 
> ---
>  linux-user/syscall.c | 23 +++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/linux-user/syscall.c b/linux-user/syscall.c
> index 2947e79dc0..e739921e86 100644
> --- a/linux-user/syscall.c
> +++ b/linux-user/syscall.c
> @@ -35,6 +35,7 @@
>  #include 
>  #include 
>  #include 
> +#include 

we don't need that include, see below

>  #include 
>  #include 
>  #include 
> @@ -8254,6 +8255,11 @@ static int host_to_target_cpu_mask(const unsigned long 
> *host_mask,
>  return 0;
>  }
>  
> +static int pivot_root(const char *new_root, const char *put_old)
> +{
> +return syscall(SYS_pivot_root, new_root, put_old);
> +}

Better to use:

#if defined(TARGET_NR_pivot_root) && defined(__NR_pivot_root)
_syscall2(int, pivot_root, const char *, new_root, const char *, put_old)
#endif

> +
>  /* This is an internal helper for do_syscall so that it is easier
>   * to have a single return point, so that actions, such as logging
>   * of syscall results, can be performed.
> @@ -13222,6 +13228,23 @@ static abi_long do_syscall1(void *cpu_env, int num, 
> abi_long arg1,
>  return ret;
>  #endif
>  
> +#if defined(TARGET_NR_pivot_root)
> +case TARGET_NR_pivot_root:
> +{
> +void *p2;
> +p = lock_user_string(arg1); /* new_root */
> +p2 = lock_user_string(arg2); /* put_old */
> +if (!p || !p2) {
> +ret = -TARGET_EFAULT;
> +} else {
> +ret = get_errno(pivot_root(p, p2));
> +}
> +unlock_user(p2, arg2, 0);
> +unlock_user(p, arg1, 0);
> +}
> +return ret;
> +#endif
> +
>  default:
>  qemu_log_mask(LOG_UNIMP, "Unsupported syscall: %d\n", num);
>  return -TARGET_ENOSYS;
> 

Thanks,
Laurent

Re: [PATCH v6 25/25] python: add tox support

[Cleber Rosa]

On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
> This is intended to be a manually run, non-CI script.
> 
> Use tox to test the linters against all python versions from 3.6 to
> 3.9. This will only work if you actually have those versions installed
> locally, but Fedora makes this easy:
> 
> > sudo dnf install python36 python37 python38 python39
> 
> Unlike the pipenv tests (make venv-check), this pulls "whichever"
> versions of the python packages, so they are unpinned and may break as
> time goes on. In the case that breakages are found, setup.cfg should be
> amended accordingly to avoid the bad dependant versions, or the code
> should be amended to work around the issue.
> 
> Signed-off-by: John Snow 
> ---
>  python/README.rst |  2 ++
>  python/.gitignore |  1 +
>  python/Makefile   |  7 ++-
>  python/setup.cfg  |  1 +
>  python/tox.ini| 13 +
>  5 files changed, 23 insertions(+), 1 deletion(-)
>  create mode 100644 python/tox.ini
> 

This works as intended for me.  A couple of notes / suggestions
for future improvements:

 * `dnf install tox` pulled all the Python versions available (I
   assume as suggestions) automatically

 * tox.ini can be folded into setup.cfg

 * a custom container image with all those Python versions may be
   handy for running both the pipenv based job (along with the
   suggestions on the previous patch) and an on-demand,
   "allow_failure" tox based CI job.

Other than those suggestions, this LGTM!

Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 


signature.asc
Description: PGP signature