Re: [PATCH v4 0/5] block-copy: use aio-task-pool

[Vladimir Sementsov-Ogievskiy]

05.05.2020 14:06, Max Reitz wrote:

On 29.04.20 15:08, Vladimir Sementsov-Ogievskiy wrote:

Hi all!

v4
01: add Max's r-b
04: move variable definition to the top of the block, add Max's r-b
05: - change error-codes in block_copy_task_run(), document them
   and be more accurate about error code in block_copy_dirty_clusters().
 - s/g_free(aio)/aio_task_pool_free(aio)/


Thanks, applied to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block



Thank you!


--
Best regards,
Vladimir

Re: backing chain & block status & filters

[Vladimir Sementsov-Ogievskiy]

01.05.2020 6:04, Andrey Shinkevich wrote:

Sounds good to me generally.
Also, we need to identify the filter by its node name when the file names of a 
node and of the filter above it are the same. And what about automatically 
generated node name for the filter? We will want to pass it to the stream 
routine.



As I understand, there should not be auto-generated node-names in blockdev era. 
All nodes are named by libvirt with full-control on them.

--
Best regards,
Vladimir

Re: [PATCH v7 0/7] reference implementation of RSS and hash report

[Jason Wang]

On 2020/5/1 下午12:01, Yuri Benditovich wrote:

Michael/Jason,

As Linux headers was updated in qemu and now include RSC/RSS/Hash 
definitions, please let me know what you prefer:
1. You apply this series as is, then I submit clean-up series that 
will remove all the redundant defines from virtio-net.c
2. I post v8 of this series with cleanup of all the redundant defines 
and also RSC ones

3. Something other



Hi Yuri:

Though I've queued this series but consider we have new headers, I think 
it's better to post v8.


Thanks

[Bug 1866870] Re: KVM Guest pauses after upgrade to Ubuntu 20.04

[Christian Ehrhardt ]

Hi Andreas,
so the only upgrade you did to trigger this for you was to bump the kernel from 
5.4.0-28.33 to 5.4.0-29.34 - nothing else? I have not (yet?) heard other 
similar reports, but it might be just too early?
At least on my system for now things still work with the new kernel like before.

I'd recommend filing a new bug, refer to this one as maybe being related and 
adding the following right away:
- kernel version (you have this here I know)
- qemu/libvirt/seabios/ovmf version (if you don't mind just attach `dpkg -l`)
- guest XML (if using libvirt) otherwise the qemu command-line
- add a cross check and report what happens with other guests configs (e.g. non 
windows, using 
  another bios as the former issue was tied to seabios, use different guest CPU 
types)
- the full /var/log/apt/history.log
- a date when you last started the VM successfully (not just 
still-had-it-running, but started it) 
  and the date when it started to fail (probably yesterday then I guess)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1866870

Title:
  KVM Guest pauses after upgrade to Ubuntu 20.04

Status in QEMU:
  Invalid
Status in qemu package in Ubuntu:
  Invalid
Status in seabios package in Ubuntu:
  Fix Released

Bug description:
  Symptom:
  Error unpausing domain: internal error: unable to execute QEMU command 
'cont': Resetting the Virtual Machine is required

  Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in 
cb_wrapper
  callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
  callback(*args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 
66, in newfn
  ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1311, in 
resume
  self._backend.resume()
File "/usr/lib/python3/dist-packages/libvirt.py", line 2174, in resume
  if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
  libvirt.libvirtError: internal error: unable to execute QEMU command 'cont': 
Resetting the Virtual Machine is required

  
  ---

  As outlined here:
  https://bugs.launchpad.net/qemu/+bug/1813165/comments/15

  After upgrade, all KVM guests are in a default pause state. Even after
  forcing them off via virsh, and restarting them the guests are paused.

  These Guests are not nested.

  A lot of diganostic information are outlined in the previous bug
  report link provided. The solution mentioned in previous report had
  been allegedly integrated into the downstream updates.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1866870/+subscriptions

RE: [Bug 1877015] [NEW] virtio only support packed ring size power of 2 between 256 and 1024

[xuan]

Thank you for your reply.  I think the code should add the handling for
packed ring size to follow the spec.

-Original Message-
From: boun...@canonical.com  On Behalf Of Yonggang Luo
Sent: Wednesday, May 6, 2020 12:32 PM
To: Ding, Xuan 
Subject: Re: [Bug 1877015] [NEW] virtio only support packed ring size power of 
2 between 256 and 1024

Maybe the virtio need to be revised?

On Wed, May 6, 2020 at 10:51 AM xuan <1877...@bugs.launchpad.net> wrote:

> Public bug reported:
>
> Issue discription：
> When QEMU starts with "-device
> virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"
>
> It raises error: Invalid rx_queue_size (= 1025), must be a power of 2 
> between 256 and 1024
>
> Analysis:
> According to virtio1.1 spec, the packed queue size value does not have 
> to be a power of 2.
>
> ** Affects: qemu
>  Importance: Undecided
>  Status: New
>
> --
> You received this bug notification because you are a member of qemu- 
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1877015
>
> Title:
>   virtio only support packed ring size power of 2 between 256 and 1024
>
> Status in QEMU:
>   New
>
> Bug description:
>   Issue discription：
>   When QEMU starts with "-device
> virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"
>
>   It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
>   between 256 and 1024
>
>   Analysis:
>   According to virtio1.1 spec, the packed queue size value does not 
> have to be a power of 2.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions
>
>

-- 
 此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1877015

Title:
  virtio only support packed ring size power of 2

Status in QEMU:
  New

Bug description:
  Issue discription：
  When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

  It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
  between 256 and 1024

  Analysis:
  According to virtio1.1 spec, the packed queue size value does not have to be 
a power of 2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1877015

Title:
  virtio only support packed ring size power of 2

Status in QEMU:
  New

Bug description:
  Issue discription：
  When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

  It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
  between 256 and 1024

  Analysis:
  According to virtio1.1 spec, the packed queue size value does not have to be 
a power of 2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions

[Bug 1877015] Re: virtio only support packed ring size power of 2

[xuan]

Thank you for your reply. I think the code should add the handling for
packed ring size to follow the spec.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1877015

Title:
  virtio only support packed ring size power of 2

Status in QEMU:
  New

Bug description:
  Issue discription：
  When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

  It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
  between 256 and 1024

  Analysis:
  According to virtio1.1 spec, the packed queue size value does not have to be 
a power of 2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions

Re: [Bug 1877015] [NEW] virtio only support packed ring size power of 2 between 256 and 1024

[Yonggang Luo]

Maybe the virtio need to be revised?

On Wed, May 6, 2020 at 10:51 AM xuan <1877...@bugs.launchpad.net> wrote:

> Public bug reported:
>
> Issue discription：
> When QEMU starts with "-device
> virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"
>
> It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
> between 256 and 1024
>
> Analysis:
> According to virtio1.1 spec, the packed queue size value does not have to
> be a power of 2.
>
> ** Affects: qemu
>  Importance: Undecided
>  Status: New
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1877015
>
> Title:
>   virtio only support packed ring size power of 2 between 256 and 1024
>
> Status in QEMU:
>   New
>
> Bug description:
>   Issue discription：
>   When QEMU starts with "-device
> virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"
>
>   It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
>   between 256 and 1024
>
>   Analysis:
>   According to virtio1.1 spec, the packed queue size value does not have
> to be a power of 2.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions
>
>

-- 
 此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo

[Bug 1860056] Re: mips binaries segfault

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1860056

Title:
  mips binaries segfault

Status in QEMU:
  Expired

Bug description:
  Hello World appears to segfault with qemu mips, on a Debian 10.0.0
  Buster amd64 host.

  Example:

  
  $ cat mips/test/hello.cpp 
  #include 
  using std::cout;

  int main() {
  cout << "Hello World!\n";
  return 0;
  }

  $ mips-linux-gnu-g++ -o hello hello.cpp && ./hello
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  Note that 64-bit MIPS and little endian 32-bit MIPS qemu work fine.
  The problem is limited to big endian 32-bit MIPS.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1860056/+subscriptions

[PATCH 0/2] migration/multifd: fix two memleaks

[Pan Nengyuan]

Fix two memleaks in multifd_send_thread/multifd_new_send_channel_async when 
error happen.

Pan Nengyuan (2):
  migration/multifd: fix memleaks in multifd_new_send_channel_async
  migration/multifd: Do error_free after migrate_set_error to avoid
memleaks

 migration/multifd.c | 5 +
 1 file changed, 5 insertions(+)

-- 
2.18.2

[PATCH 2/2] migration/multifd: Do error_free after migrate_set_error to avoid memleaks

[Pan Nengyuan]

When error happen in multifd_send_thread, it use error_copy to set migrate 
error in
multifd_send_terminate_threads(). We should call error_free after it.

Similarly, fix another two places in multifd_recv_thread/multifd_save_cleanup.

The leak stack:
Direct leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7f781af07cf0 in calloc (/lib64/libasan.so.5+0xefcf0)
#1 0x7f781a2ce22d in g_malloc0 (/lib64/libglib-2.0.so.0+0x5322d)
#2 0x55ee1d075c17 in error_setv /mnt/sdb/backup/qemu/util/error.c:61
#3 0x55ee1d076464 in error_setg_errno_internal 
/mnt/sdb/backup/qemu/util/error.c:109
#4 0x55ee1cef066e in qio_channel_socket_writev 
/mnt/sdb/backup/qemu/io/channel-socket.c:569
#5 0x55ee1cee806b in qio_channel_writev 
/mnt/sdb/backup/qemu/io/channel.c:207
#6 0x55ee1cee806b in qio_channel_writev_all 
/mnt/sdb/backup/qemu/io/channel.c:171
#7 0x55ee1cee8248 in qio_channel_write_all 
/mnt/sdb/backup/qemu/io/channel.c:257
#8 0x55ee1ca12c9a in multifd_send_thread 
/mnt/sdb/backup/qemu/migration/multifd.c:657
#9 0x55ee1d0607fc in qemu_thread_start 
/mnt/sdb/backup/qemu/util/qemu-thread-posix.c:519
#10 0x7f78159ae2dd in start_thread (/lib64/libpthread.so.0+0x82dd)
#11 0x7f78156df4b2 in __GI___clone (/lib64/libc.so.6+0xfc4b2)

Indirect leak of 52 byte(s) in 1 object(s) allocated from:
#0 0x7f781af07f28 in __interceptor_realloc (/lib64/libasan.so.5+0xeff28)
#1 0x7f78156f07d9 in __GI___vasprintf_chk (/lib64/libc.so.6+0x10d7d9)
#2 0x7f781a30ea6c in g_vasprintf (/lib64/libglib-2.0.so.0+0x93a6c)
#3 0x7f781a2e7cd0 in g_strdup_vprintf (/lib64/libglib-2.0.so.0+0x6ccd0)
#4 0x7f781a2e7d8c in g_strdup_printf (/lib64/libglib-2.0.so.0+0x6cd8c)
#5 0x55ee1d075c86 in error_setv /mnt/sdb/backup/qemu/util/error.c:65
#6 0x55ee1d076464 in error_setg_errno_internal 
/mnt/sdb/backup/qemu/util/error.c:109
#7 0x55ee1cef066e in qio_channel_socket_writev 
/mnt/sdb/backup/qemu/io/channel-socket.c:569
#8 0x55ee1cee806b in qio_channel_writev 
/mnt/sdb/backup/qemu/io/channel.c:207
#9 0x55ee1cee806b in qio_channel_writev_all 
/mnt/sdb/backup/qemu/io/channel.c:171
#10 0x55ee1cee8248 in qio_channel_write_all 
/mnt/sdb/backup/qemu/io/channel.c:257
#11 0x55ee1ca12c9a in multifd_send_thread 
/mnt/sdb/backup/qemu/migration/multifd.c:657
#12 0x55ee1d0607fc in qemu_thread_start 
/mnt/sdb/backup/qemu/util/qemu-thread-posix.c:519
#13 0x7f78159ae2dd in start_thread (/lib64/libpthread.so.0+0x82dd)
#14 0x7f78156df4b2 in __GI___clone (/lib64/libc.so.6+0xfc4b2)

Reported-by: Euler Robot 
Signed-off-by: Pan Nengyuan 
---
 migration/multifd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/migration/multifd.c b/migration/multifd.c
index 197d59294a..35ae3180d2 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -550,6 +550,7 @@ void multifd_save_cleanup(void)
 multifd_send_state->ops->send_cleanup(p, _err);
 if (local_err) {
 migrate_set_error(migrate_get_current(), local_err);
+error_free(local_err);
 }
 }
 qemu_sem_destroy(_send_state->channels_ready);
@@ -688,6 +689,7 @@ out:
 if (local_err) {
 trace_multifd_send_error(p->id);
 multifd_send_terminate_threads(local_err);
+error_free(local_err);
 }
 
 /*
@@ -965,6 +967,7 @@ static void *multifd_recv_thread(void *opaque)
 
 if (local_err) {
 multifd_recv_terminate_threads(local_err);
+error_free(local_err);
 }
 qemu_mutex_lock(>mutex);
 p->running = false;
-- 
2.18.2

[PATCH 1/2] migration/multifd: fix memleaks in multifd_new_send_channel_async

[Pan Nengyuan]

When error happen in multifd_new_send_channel_async, 'sioc' will not be used
to create the multifd_send_thread. Let's free it to avoid a memleak. And also
do error_free after migrate_set_error() to avoid another leak in the same place.

The leak stack:
Direct leak of 2880 byte(s) in 8 object(s) allocated from:
#0 0x7f20b5118ae8 in __interceptor_malloc (/lib64/libasan.so.5+0xefae8)
#1 0x7f20b44df1d5 in g_malloc (/lib64/libglib-2.0.so.0+0x531d5)
#2 0x564133bce18b in object_new_with_type 
/mnt/sdb/backup/qemu/qom/object.c:683
#3 0x564133eea950 in qio_channel_socket_new 
/mnt/sdb/backup/qemu/io/channel-socket.c:56
#4 0x5641339cfe4f in socket_send_channel_create 
/mnt/sdb/backup/qemu/migration/socket.c:37
#5 0x564133a10328 in multifd_save_setup 
/mnt/sdb/backup/qemu/migration/multifd.c:772
#6 0x5641339cebed in migrate_fd_connect 
/mnt/sdb/backup/qemu/migration/migration.c:3530
#7 0x5641339d15e4 in migration_channel_connect 
/mnt/sdb/backup/qemu/migration/channel.c:92
#8 0x5641339cf5b7 in socket_outgoing_migration 
/mnt/sdb/backup/qemu/migration/socket.c:108

Direct leak of 384 byte(s) in 8 object(s) allocated from:
#0 0x7f20b5118cf0 in calloc (/lib64/libasan.so.5+0xefcf0)
#1 0x7f20b44df22d in g_malloc0 (/lib64/libglib-2.0.so.0+0x5322d)
#2 0x56413406fc17 in error_setv /mnt/sdb/backup/qemu/util/error.c:61
#3 0x564134070464 in error_setg_errno_internal 
/mnt/sdb/backup/qemu/util/error.c:109
#4 0x5641340851be in inet_connect_addr 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:379
#5 0x5641340851be in inet_connect_saddr 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:458
#6 0x5641340870ab in socket_connect 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:1105
#7 0x564133eeaabf in qio_channel_socket_connect_sync 
/mnt/sdb/backup/qemu/io/channel-socket.c:145
#8 0x564133eeabf5 in qio_channel_socket_connect_worker 
/mnt/sdb/backup/qemu/io/channel-socket.c:168

Indirect leak of 360 byte(s) in 8 object(s) allocated from:
#0 0x7f20b5118ae8 in __interceptor_malloc (/lib64/libasan.so.5+0xefae8)
#1 0x7f20af901817 in __GI___vasprintf_chk (/lib64/libc.so.6+0x10d817)
#2 0x7f20b451fa6c in g_vasprintf (/lib64/libglib-2.0.so.0+0x93a6c)
#3 0x7f20b44f8cd0 in g_strdup_vprintf (/lib64/libglib-2.0.so.0+0x6ccd0)
#4 0x7f20b44f8d8c in g_strdup_printf (/lib64/libglib-2.0.so.0+0x6cd8c)
#5 0x56413406fc86 in error_setv /mnt/sdb/backup/qemu/util/error.c:65
#6 0x564134070464 in error_setg_errno_internal 
/mnt/sdb/backup/qemu/util/error.c:109
#7 0x5641340851be in inet_connect_addr 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:379
#8 0x5641340851be in inet_connect_saddr 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:458
#9 0x5641340870ab in socket_connect 
/mnt/sdb/backup/qemu/util/qemu-sockets.c:1105
#10 0x564133eeaabf in qio_channel_socket_connect_sync 
/mnt/sdb/backup/qemu/io/channel-socket.c:145
#11 0x564133eeabf5 in qio_channel_socket_connect_worker 
/mnt/sdb/backup/qemu/io/channel-socket.c:168

Reported-by: Euler Robot 
Signed-off-by: Pan Nengyuan 
---
 migration/multifd.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/migration/multifd.c b/migration/multifd.c
index cb6a4a3ab8..197d59294a 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -727,6 +727,8 @@ static void multifd_new_send_channel_async(QIOTask *task, 
gpointer opaque)
  * its status.
  */
 p->quit = true;
+object_unref(OBJECT(sioc));
+error_free(local_err);
 } else {
 p->c = QIO_CHANNEL(sioc);
 qio_channel_set_delay(p->c, false);
-- 
2.18.2

[Bug 1877015] Re: virtio only support packed ring size power of 2

[xuan]

** Summary changed:

- virtio only support packed ring size power of 2 between 256 and 1024
+ virtio only support packed ring size power of 2

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1877015

Title:
  virtio only support packed ring size power of 2

Status in QEMU:
  New

Bug description:
  Issue discription：
  When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

  It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
  between 256 and 1024

  Analysis:
  According to virtio1.1 spec, the packed queue size value does not have to be 
a power of 2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions

[Bug 1877015] [NEW] virtio only support packed ring size power of 2 between 256 and 1024

[xuan]

Public bug reported:

Issue discription：
When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
between 256 and 1024

Analysis:
According to virtio1.1 spec, the packed queue size value does not have to be a 
power of 2.

** Affects: qemu
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1877015

Title:
  virtio only support packed ring size power of 2 between 256 and 1024

Status in QEMU:
  New

Bug description:
  Issue discription：
  When QEMU starts with "-device 
virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1025,tx_queue_size=1025,mq=on,vectors=15,packed=on"

  It raises error: Invalid rx_queue_size (= 1025), must be a power of 2
  between 256 and 1024

  Analysis:
  According to virtio1.1 spec, the packed queue size value does not have to be 
a power of 2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1877015/+subscriptions

Re: [PATCH v2 1/1] target-ppc: fix rlwimi, rlwinm, rlwnm for Clang-9

[David Gibson]

On Tue, May 05, 2020 at 02:38:17PM -0400, Daniele Buono wrote:
> Starting with Clang v9, -Wtype-limits is implemented and triggers a
> few "result of comparison is always true" errors when compiling PPC32
> targets.
> 
> The comparisons seem to be necessary only on PPC64, since the
> else branch in PPC32 only has a "g_assert_not_reached();" in all cases.
> 
> This patch restructures the code so that the actual if/else is done on a
> local flag variable, that is set accordingly for PPC64, and always
> true for PPC32.
> 
> Signed-off-by: Daniele Buono 

That's a bit nicer, thanks.  Applied to ppc-for-5.1 in place of the
earlier version.

> ---
>  target/ppc/translate.c | 24 +---
>  1 file changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/target/ppc/translate.c b/target/ppc/translate.c
> index 807d14faaa..338529879f 100644
> --- a/target/ppc/translate.c
> +++ b/target/ppc/translate.c
> @@ -1882,6 +1882,7 @@ static void gen_rlwimi(DisasContext *ctx)
>  tcg_gen_deposit_tl(t_ra, t_ra, t_rs, sh, me - mb + 1);
>  } else {
>  target_ulong mask;
> +bool mask_in_32b = true;
>  TCGv t1;
>  
>  #if defined(TARGET_PPC64)
> @@ -1890,8 +1891,13 @@ static void gen_rlwimi(DisasContext *ctx)
>  #endif
>  mask = MASK(mb, me);
>  
> +#if defined(TARGET_PPC64)
> +if (mask > 0xu) {
> +mask_in_32b = false;
> +}
> +#endif
>  t1 = tcg_temp_new();
> -if (mask <= 0xu) {
> +if (mask_in_32b) {
>  TCGv_i32 t0 = tcg_temp_new_i32();
>  tcg_gen_trunc_tl_i32(t0, t_rs);
>  tcg_gen_rotli_i32(t0, t0, sh);
> @@ -1933,12 +1939,18 @@ static void gen_rlwinm(DisasContext *ctx)
>  tcg_gen_extract_tl(t_ra, t_rs, rsh, len);
>  } else {
>  target_ulong mask;
> +bool mask_in_32b = true;
>  #if defined(TARGET_PPC64)
>  mb += 32;
>  me += 32;
>  #endif
>  mask = MASK(mb, me);
> -if (mask <= 0xu) {
> +#if defined(TARGET_PPC64)
> +if (mask > 0xu) {
> +mask_in_32b = false;
> +}
> +#endif
> +if (mask_in_32b) {
>  if (sh == 0) {
>  tcg_gen_andi_tl(t_ra, t_rs, mask);
>  } else {
> @@ -1973,6 +1985,7 @@ static void gen_rlwnm(DisasContext *ctx)
>  uint32_t mb = MB(ctx->opcode);
>  uint32_t me = ME(ctx->opcode);
>  target_ulong mask;
> +bool mask_in_32b = true;
>  
>  #if defined(TARGET_PPC64)
>  mb += 32;
> @@ -1980,7 +1993,12 @@ static void gen_rlwnm(DisasContext *ctx)
>  #endif
>  mask = MASK(mb, me);
>  
> -if (mask <= 0xu) {
> +#if defined(TARGET_PPC64)
> +if (mask > 0xu) {
> +mask_in_32b = false;
> +}
> +#endif
> +if (mask_in_32b) {
>  TCGv_i32 t0 = tcg_temp_new_i32();
>  TCGv_i32 t1 = tcg_temp_new_i32();
>  tcg_gen_trunc_tl_i32(t0, t_rb);

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [PATCH for-5.1 V3 0/7] mips: Add Loongson-3 machine support (with KVM)

[Huacai Chen]

Hi, Aleksandar,

On Tue, May 5, 2020 at 6:12 PM Aleksandar Markovic
 wrote:
>
>
>
> уторак, 05. мај 2020., chen huacai  је написао/ла:
>>
>> Hi, Aleksandar,
>>
>> On Sun, May 3, 2020 at 6:50 PM Aleksandar Markovic
>>  wrote:
>> >
>> > нед, 3. мај 2020. у 12:21 Huacai Chen  је 
>> > написао/ла:
>> > >
>> > > Loongson-3 CPU family include Loongson-3A R1/R2/R3/R4 and Loongson-3B
>> > > R1/R2. Loongson-3A R1 is the oldest and its ISA is the smallest, while
>> > > Loongson-3A R4 is the newest and its ISA is almost the superset of all
>> > > others. To reduce complexity, in QEMU we just define two CPU types:
>> > >
>> > > 1, "Loongson-3A1000" CPU which is corresponding to Loongson-3A R1. It is
>> > >suitable for TCG because Loongson-3A R1 has fewest ASE.
>> > > 2, "Loongson-3A4000" CPU which is corresponding to Loongson-3A R4. It is
>> > >suitable for KVM because Loongson-3A R4 has the VZ ASE.
>> > >
>> >
>> > Huacai, thanks for putting together v3, which is a little better than v2, 
>> > and
>> > thanks for addressing my previous suggestions.
>> >
>> > Now, give us some time to digest new data on Loongson3.  We will
>> > respond, but it won't happen immediately, which is, you'd agree,
>> > reasonable. Just be patient.
>> >
>> > But again, in general, I salute your efforts very much!
>> >
>> > Yours, Aleksandar
>> I'm sorry for this late response because I have done many tests to
>> reproduce the problem reported at
>> https://patchew.org/QEMU/1588501221-1205-1-git-send-email-che...@lemote.com/,
>> but I don't have such a failure...
>>
>> What I have done:
>> 1, "make check" on MIPS64 platform (distro is Fedora28 for Loongson);
>> 2, "make check" on X86_64 platform (distro is RHEL8);
>> 3, "make docker-test-quick@centos7 SHOW_ENV=1 NETWORK=1" on X86_64
>> platform (distro is RHEL8);
>> 4, "make docker-test-quick@centos7 SHOW_ENV=1 J=n NETWORK=1" on X86_64
>> platform (distro is RHEL8 and I've tried n=2,3,414);
>>
>> I always get the same result:
>> Not run: 259
>> Passed all 117 iotests
>>
>> And, it seems that my patchset doesn't touch anything about iotests,
>> so I don't know why the build test fails on iotests 192 (Maybe your
>> build test has the same problem without my patches).
>>
>
> From time to time, there is some instability in our automatic iotests. You 
> shouldn't bother too much about it, of course you retest in your 
> environments, that is good. But, in all likelyhood, your patchset doesn't 
> really have anything to do with the reported iotest failure.

Thank you for your help, and here is another question: Could
"target/mips: Support variable page size" series be merged now? This
Loongson-3 series depend on variable page size logically.

>
> Truly yours,
> Aleksandar
>
>
>>
>> P.S.: I have found a problem that my patchset has a build failure with
>> CONFIG_KVM=n, but this is another problem and I will send V4 to fix it
>> (after collecting all problems in V3).
>>
>>
>> >
>> > > Loongson-3 lacks English documents. I've tried to translated them with
>> > > translate.google.com, and the machine translated documents (together
>> > > with their original Chinese versions) are available here.
>> > >
>> > > Loongson-3A R1 (Loongson-3A1000)
>> > > User Manual Part 1:
>> > > http://ftp.godson.ac.cn/lemote/3A1000_p1.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A1000_processor_user_manual_P1.pdf
>> > >  (Chinese Version)
>> > > User Manual Part 2:
>> > > http://ftp.godson.ac.cn/lemote/3A1000_p2.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A1000_processor_user_manual_P2.pdf
>> > >  (Chinese Version)
>> > >
>> > > Loongson-3A R2 (Loongson-3A2000)
>> > > User Manual Part 1:
>> > > http://ftp.godson.ac.cn/lemote/3A2000_p1.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A2000_user1.pdf (Chinese Version)
>> > > User Manual Part 2:
>> > > http://ftp.godson.ac.cn/lemote/3A2000_p2.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A2000_user2.pdf (Chinese Version)
>> > >
>> > > Loongson-3A R3 (Loongson-3A3000)
>> > > User Manual Part 1:
>> > > http://ftp.godson.ac.cn/lemote/3A3000_p1.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A3000_3B3000usermanual1.pdf 
>> > > (Chinese Version)
>> > > User Manual Part 2:
>> > > http://ftp.godson.ac.cn/lemote/3A3000_p2.pdf
>> > > http://ftp.godson.ac.cn/lemote/Loongson3A3000_3B3000usermanual2.pdf 
>> > > (Chinese Version)
>> > >
>> > > Loongson-3A R4 (Loongson-3A4000)
>> > > User Manual Part 1:
>> > > http://ftp.godson.ac.cn/lemote/3A4000_p1.pdf
>> > > http://ftp.godson.ac.cn/lemote/3A4000user.pdf (Chinese Version)
>> > > User Manual Part 2:
>> > > I'm sorry that it is unavailable now.
>> > >
>> > > We are preparing to add QEMU's Loongson-3 support. MIPS VZ extension is
>> > > fully supported in Loongson-3A R4+, so we at first add QEMU/KVM support
>> > > in this series. And the next series will add QEMU/TCG support (it will
>> > > emulate Loongson-3A R1).
>> > >
>> > > We already have a full functional Linux kernel (based on Linux-5.4.x LTS
>> > 

Re: [PATCH] riscv: Change the default behavior if no -bios option is specified

[Bin Meng]

Hi Alistair,

On Wed, May 6, 2020 at 6:37 AM Alistair Francis  wrote:
>
> On Tue, May 5, 2020 at 1:34 PM Alistair Francis  wrote:
> >
> > On Fri, May 1, 2020 at 5:21 AM Bin Meng  wrote:
> > >
> > > From: Bin Meng 
> > >
> > > Per QEMU deprecated doc, QEMU 4.1 introduced support for the -bios
> > > option in QEMU for RISC-V for the virt machine and sifive_u machine.
> > > The default behavior has been that QEMU does not automatically load
> > > any firmware if no -bios option is included.
> > >
> > > Now 2 releases passed, it's time to change the default behavior to
> > > load the default OpenSBI firmware automatically. The firmware is
> > > included with the QEMU release and no user interaction is required.
> > > All a user needs to do is specify the kernel they want to boot with
> > > the -kernel option.
> > >
> > > Signed-off-by: Bin Meng 
> >
> > Thanks!
> >
> > Reviewed-by: Alistair Francis 
> >
> > Applied to the RISC-V tree.
>
> This fails `make check`
>
> qemu-system-riscv64: Unable to load the RISC-V firmware
> "opensbi-riscv64-spike-fw_jump.elf"
> Broken pipe
> /scratch/alistair/software/qemu/tests/qtest/libqtest.c:166:
> kill_qemu() tried to terminate QEMU process but encountered exit
> status 1 (expected 0)
> ERROR - too few tests run (expected 7, got 2)
> make: *** [/scratch/alistair/software/qemu/tests/Makefile.include:637:
> check-qtest-riscv64] Error 1
>

Please apply this patch to fix the "make check" as well.

[5/5] riscv: Suppress the error report for QEMU testing with
riscv_find_firmware()
http://patchwork.ozlabs.org/project/qemu-devel/patch/1588348254-7241-6-git-send-email-bmeng...@gmail.com/

Regards,
Bin

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
6d65beb qom: Drop @errp parameter of object_property_del()
458cedb spapr_pci: Drop some dead error handling
7ffb546 qdev: Unrealize must not fail
bfab387 Drop more @errp parameters after previous commit
093140a qom: Drop parameter @errp of object_property_add() & friends
7802bbe qdev: Clean up qdev_connect_gpio_out_named()
674837c hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
0c51050 e1000: Don't run e1000_instance_init() twice
0b08b0f hw/isa/superio: Make the components QOM children
ba292c0 s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
059650f tests/check-qom-proplist: Improve iterator coverage
54d70f2 qom: Drop object_property_set_description() parameter @errp
e4c477d qom: Make all the object_property_add_FOO() return the property
40d45ef qom: Drop convenience method object_property_get_uint16List()
544020b qom: Simplify object_property_get_enum()
c3b9c57 qom: Drop object_property_del_child()'s unused parameter @errp
2d06cf9 qom: Clean up inconsistent use of gchar * vs. char *
452b735 qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 452b735640b3 (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit 2d06cf9fbb63 (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit c3b9c571abbb (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit 544020b6690d (qom: Simplify object_property_get_enum())
5/18 Checking commit 40d45efa4a1f (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit e4c477d06477 (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit 54d70f2a2aa2 (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit 059650f6eb73 (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit ba292c0d650a (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit 0b08b0fd9725 (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit 0c51050ca568 (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit 674837c074d8 (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit 7802bbefdfb9 (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit 093140aa2723 (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit bfab387eb1fe (Drop more @errp parameters after previous 
commit)
16/18 Checking commit 7ffb54686f74 (qdev: Unrealize must not fail)
17/18 Checking commit 458cedbe9c6e (spapr_pci: Drop some dead error handling)
18/18 Checking commit 6d65beb23534 (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[PATCH v1 2/2] target/riscv: Drop support for ISA spec version 1.09.1

[Alistair Francis]

The RISC-V ISA spec version 1.09.1 has been deprecated in QEMU since
4.1. It's not commonly used so let's remove support for it.

Signed-off-by: Alistair Francis 
---
 target/riscv/cpu.c| 30 ---
 target/riscv/cpu.h|  8 --
 target/riscv/csr.c| 82 ---
 .../riscv/insn_trans/trans_privileged.inc.c   |  6 --
 tests/qtest/machine-none-test.c   |  4 +-
 5 files changed, 19 insertions(+), 111 deletions(-)

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 059d71f2c7..eeb91f8513 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -135,16 +135,6 @@ static void riscv_base32_cpu_init(Object *obj)
 set_misa(env, 0);
 }
 
-static void rv32gcsu_priv1_09_1_cpu_init(Object *obj)
-{
-CPURISCVState *env = _CPU(obj)->env;
-set_misa(env, RV32 | RVI | RVM | RVA | RVF | RVD | RVC | RVS | RVU);
-set_priv_version(env, PRIV_VERSION_1_09_1);
-set_resetvec(env, DEFAULT_RSTVEC);
-set_feature(env, RISCV_FEATURE_MMU);
-set_feature(env, RISCV_FEATURE_PMP);
-}
-
 static void rv32gcsu_priv1_10_0_cpu_init(Object *obj)
 {
 CPURISCVState *env = _CPU(obj)->env;
@@ -182,16 +172,6 @@ static void riscv_base64_cpu_init(Object *obj)
 set_misa(env, 0);
 }
 
-static void rv64gcsu_priv1_09_1_cpu_init(Object *obj)
-{
-CPURISCVState *env = _CPU(obj)->env;
-set_misa(env, RV64 | RVI | RVM | RVA | RVF | RVD | RVC | RVS | RVU);
-set_priv_version(env, PRIV_VERSION_1_09_1);
-set_resetvec(env, DEFAULT_RSTVEC);
-set_feature(env, RISCV_FEATURE_MMU);
-set_feature(env, RISCV_FEATURE_PMP);
-}
-
 static void rv64gcsu_priv1_10_0_cpu_init(Object *obj)
 {
 CPURISCVState *env = _CPU(obj)->env;
@@ -388,8 +368,6 @@ static void riscv_cpu_realize(DeviceState *dev, Error 
**errp)
 priv_version = PRIV_VERSION_1_11_0;
 } else if (!g_strcmp0(cpu->cfg.priv_spec, "v1.10.0")) {
 priv_version = PRIV_VERSION_1_10_0;
-} else if (!g_strcmp0(cpu->cfg.priv_spec, "v1.9.1")) {
-priv_version = PRIV_VERSION_1_09_1;
 } else {
 error_setg(errp,
"Unsupported privilege spec version '%s'",
@@ -621,18 +599,10 @@ static const TypeInfo riscv_cpu_type_infos[] = {
 DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E31,   rv32imacu_nommu_cpu_init),
 DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E34,   rv32imafcu_nommu_cpu_init),
 DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U34,   rv32gcsu_priv1_10_0_cpu_init),
-/* Depreacted */
-DEFINE_CPU(TYPE_RISCV_CPU_RV32IMACU_NOMMU,  rv32imacu_nommu_cpu_init),
-DEFINE_CPU(TYPE_RISCV_CPU_RV32GCSU_V1_09_1, rv32gcsu_priv1_09_1_cpu_init),
-DEFINE_CPU(TYPE_RISCV_CPU_RV32GCSU_V1_10_0, rv32gcsu_priv1_10_0_cpu_init)
 #elif defined(TARGET_RISCV64)
 DEFINE_CPU(TYPE_RISCV_CPU_BASE64,   riscv_base64_cpu_init),
 DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E51,   rv64imacu_nommu_cpu_init),
 DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U54,   rv64gcsu_priv1_10_0_cpu_init),
-/* Deprecated */
-DEFINE_CPU(TYPE_RISCV_CPU_RV64IMACU_NOMMU,  rv64imacu_nommu_cpu_init),
-DEFINE_CPU(TYPE_RISCV_CPU_RV64GCSU_V1_09_1, rv64gcsu_priv1_09_1_cpu_init),
-DEFINE_CPU(TYPE_RISCV_CPU_RV64GCSU_V1_10_0, rv64gcsu_priv1_10_0_cpu_init)
 #endif
 };
 
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index d0e7f5b9c5..c022539012 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -40,13 +40,6 @@
 #define TYPE_RISCV_CPU_SIFIVE_E51   RISCV_CPU_TYPE_NAME("sifive-e51")
 #define TYPE_RISCV_CPU_SIFIVE_U34   RISCV_CPU_TYPE_NAME("sifive-u34")
 #define TYPE_RISCV_CPU_SIFIVE_U54   RISCV_CPU_TYPE_NAME("sifive-u54")
-/* Deprecated */
-#define TYPE_RISCV_CPU_RV32IMACU_NOMMU  RISCV_CPU_TYPE_NAME("rv32imacu-nommu")
-#define TYPE_RISCV_CPU_RV32GCSU_V1_09_1 RISCV_CPU_TYPE_NAME("rv32gcsu-v1.9.1")
-#define TYPE_RISCV_CPU_RV32GCSU_V1_10_0 RISCV_CPU_TYPE_NAME("rv32gcsu-v1.10.0")
-#define TYPE_RISCV_CPU_RV64IMACU_NOMMU  RISCV_CPU_TYPE_NAME("rv64imacu-nommu")
-#define TYPE_RISCV_CPU_RV64GCSU_V1_09_1 RISCV_CPU_TYPE_NAME("rv64gcsu-v1.9.1")
-#define TYPE_RISCV_CPU_RV64GCSU_V1_10_0 RISCV_CPU_TYPE_NAME("rv64gcsu-v1.10.0")
 
 #define RV32 ((target_ulong)1 << (TARGET_LONG_BITS - 2))
 #define RV64 ((target_ulong)2 << (TARGET_LONG_BITS - 2))
@@ -80,7 +73,6 @@ enum {
 RISCV_FEATURE_MISA
 };
 
-#define PRIV_VERSION_1_09_1 0x00010901
 #define PRIV_VERSION_1_10_0 0x00011000
 #define PRIV_VERSION_1_11_0 0x00011100
 
diff --git a/target/riscv/csr.c b/target/riscv/csr.c
index 11d184cd16..df3498b24f 100644
--- a/target/riscv/csr.c
+++ b/target/riscv/csr.c
@@ -58,31 +58,11 @@ static int ctr(CPURISCVState *env, int csrno)
 #if !defined(CONFIG_USER_ONLY)
 CPUState *cs = env_cpu(env);
 RISCVCPU *cpu = RISCV_CPU(cs);
-uint32_t ctr_en = ~0u;
 
 if (!cpu->cfg.ext_counters) {
 /* The Counters extensions is not enabled */
 return -1;
 }
-
-/*
- * The counters are always 

[PATCH v1 1/2] hw/riscv: spike: Remove deprecated ISA specific machines

[Alistair Francis]

The ISA specific Spike machines have  been deprecated in QEMU since 4.1,
let's finally remove them.

Signed-off-by: Alistair Francis 
---
 hw/riscv/spike.c | 217 ---
 include/hw/riscv/spike.h |   6 +-
 2 files changed, 2 insertions(+), 221 deletions(-)

diff --git a/hw/riscv/spike.c b/hw/riscv/spike.c
index d0c4843712..7bbbdb5036 100644
--- a/hw/riscv/spike.c
+++ b/hw/riscv/spike.c
@@ -257,221 +257,6 @@ static void spike_board_init(MachineState *machine)
 false);
 }
 
-static void spike_v1_10_0_board_init(MachineState *machine)
-{
-const struct MemmapEntry *memmap = spike_memmap;
-
-SpikeState *s = g_new0(SpikeState, 1);
-MemoryRegion *system_memory = get_system_memory();
-MemoryRegion *main_mem = g_new(MemoryRegion, 1);
-MemoryRegion *mask_rom = g_new(MemoryRegion, 1);
-int i;
-unsigned int smp_cpus = machine->smp.cpus;
-
-if (!qtest_enabled()) {
-info_report("The Spike v1.10.0 machine has been deprecated. "
-"Please use the generic spike machine and specify the ISA "
-"versions using -cpu.");
-}
-
-/* Initialize SOC */
-object_initialize_child(OBJECT(machine), "soc", >soc, sizeof(s->soc),
-TYPE_RISCV_HART_ARRAY, _abort, NULL);
-object_property_set_str(OBJECT(>soc), SPIKE_V1_10_0_CPU, "cpu-type",
-_abort);
-object_property_set_int(OBJECT(>soc), smp_cpus, "num-harts",
-_abort);
-object_property_set_bool(OBJECT(>soc), true, "realized",
-_abort);
-
-/* register system main memory (actual RAM) */
-memory_region_init_ram(main_mem, NULL, "riscv.spike.ram",
-   machine->ram_size, _fatal);
-memory_region_add_subregion(system_memory, memmap[SPIKE_DRAM].base,
-main_mem);
-
-/* create device tree */
-create_fdt(s, memmap, machine->ram_size, machine->kernel_cmdline);
-
-/* boot rom */
-memory_region_init_rom(mask_rom, NULL, "riscv.spike.mrom",
-   memmap[SPIKE_MROM].size, _fatal);
-memory_region_add_subregion(system_memory, memmap[SPIKE_MROM].base,
-mask_rom);
-
-if (machine->kernel_filename) {
-riscv_load_kernel(machine->kernel_filename, htif_symbol_callback);
-}
-
-/* reset vector */
-uint32_t reset_vec[8] = {
-0x0297,  /* 1:  auipc  t0, %pcrel_hi(dtb) */
-0x02028593,  /* addi   a1, t0, %pcrel_lo(1b) */
-0xf1402573,  /* csrr   a0, mhartid  */
-#if defined(TARGET_RISCV32)
-0x0182a283,  /* lw t0, 24(t0) */
-#elif defined(TARGET_RISCV64)
-0x0182b283,  /* ld t0, 24(t0) */
-#endif
-0x00028067,  /* jr t0 */
-0x,
-memmap[SPIKE_DRAM].base, /* start: .dword DRAM_BASE */
-0x,
- /* dtb: */
-};
-
-/* copy in the reset vector in little_endian byte order */
-for (i = 0; i < sizeof(reset_vec) >> 2; i++) {
-reset_vec[i] = cpu_to_le32(reset_vec[i]);
-}
-rom_add_blob_fixed_as("mrom.reset", reset_vec, sizeof(reset_vec),
-  memmap[SPIKE_MROM].base, _space_memory);
-
-/* copy in the device tree */
-if (fdt_pack(s->fdt) || fdt_totalsize(s->fdt) >
-memmap[SPIKE_MROM].size - sizeof(reset_vec)) {
-error_report("not enough space to store device-tree");
-exit(1);
-}
-qemu_fdt_dumpdtb(s->fdt, fdt_totalsize(s->fdt));
-rom_add_blob_fixed_as("mrom.fdt", s->fdt, fdt_totalsize(s->fdt),
-  memmap[SPIKE_MROM].base + sizeof(reset_vec),
-  _space_memory);
-
-/* initialize HTIF using symbols found in load_kernel */
-htif_mm_init(system_memory, mask_rom, >soc.harts[0].env, serial_hd(0));
-
-/* Core Local Interruptor (timer and IPI) */
-sifive_clint_create(memmap[SPIKE_CLINT].base, memmap[SPIKE_CLINT].size,
-smp_cpus, SIFIVE_SIP_BASE, SIFIVE_TIMECMP_BASE, SIFIVE_TIME_BASE,
-false);
-}
-
-static void spike_v1_09_1_board_init(MachineState *machine)
-{
-const struct MemmapEntry *memmap = spike_memmap;
-
-SpikeState *s = g_new0(SpikeState, 1);
-MemoryRegion *system_memory = get_system_memory();
-MemoryRegion *main_mem = g_new(MemoryRegion, 1);
-MemoryRegion *mask_rom = g_new(MemoryRegion, 1);
-int i;
-unsigned int smp_cpus = machine->smp.cpus;
-
-if (!qtest_enabled()) {
-info_report("The Spike v1.09.1 machine has been deprecated. "
-"Please use the generic spike machine and specify the ISA "
-"versions using -cpu.");
-}
-
-/* Initialize SOC */
-object_initialize_child(OBJECT(machine), "soc", >soc, sizeof(s->soc),
-  

[PATCH v1 0/2] RTISC-V: Remove deprecated ISA, CPUs and machines

[Alistair Francis]

Alistair Francis (2):
  hw/riscv: spike: Remove deprecated ISA specific machines
  target/riscv: Drop support for ISA spec version 1.09.1

 hw/riscv/spike.c  | 217 --
 include/hw/riscv/spike.h  |   6 +-
 target/riscv/cpu.c|  30 ---
 target/riscv/cpu.h|   8 -
 target/riscv/csr.c|  82 ++-
 .../riscv/insn_trans/trans_privileged.inc.c   |   6 -
 tests/qtest/machine-none-test.c   |   4 +-
 7 files changed, 21 insertions(+), 332 deletions(-)

-- 
2.26.2

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
0bfeda6 qom: Drop @errp parameter of object_property_del()
fce53a6 spapr_pci: Drop some dead error handling
bf4fa1d qdev: Unrealize must not fail
c984f35 Drop more @errp parameters after previous commit
3217176 qom: Drop parameter @errp of object_property_add() & friends
e43f7cf qdev: Clean up qdev_connect_gpio_out_named()
25ed6da hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
2b01f96 e1000: Don't run e1000_instance_init() twice
0859a4a hw/isa/superio: Make the components QOM children
0ad9369 s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
a7bcd4a tests/check-qom-proplist: Improve iterator coverage
a89bf38 qom: Drop object_property_set_description() parameter @errp
2c9507b qom: Make all the object_property_add_FOO() return the property
caeed6f qom: Drop convenience method object_property_get_uint16List()
670a5b1 qom: Simplify object_property_get_enum()
4e0ef1e qom: Drop object_property_del_child()'s unused parameter @errp
3305770 qom: Clean up inconsistent use of gchar * vs. char *
237690a qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 237690a0fd11 (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit 33057701be2e (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit 4e0ef1e4e32d (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit 670a5b17a0f5 (qom: Simplify object_property_get_enum())
5/18 Checking commit caeed6fde79c (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit 2c9507b8735b (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit a89bf3817999 (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit a7bcd4a69d42 (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit 0ad93695c816 (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit 0859a4ae84dc (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit 2b01f963380b (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit 25ed6da2c511 (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit e43f7cf1877d (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit 321717690e42 (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit c984f35ccd32 (Drop more @errp parameters after previous 
commit)
16/18 Checking commit bf4fa1d40108 (qdev: Unrealize must not fail)
17/18 Checking commit fce53a6de430 (spapr_pci: Drop some dead error handling)
18/18 Checking commit 0bfeda6594be (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[Bug 1805256] Re: qemu-img hangs on rcu_call_ready_event logic in Aarch64 when converting images

[dann frazier]

fyi, I backported that fix also to focal/groovy and eoan, and with those
builds. On my test systems the hang reliable occurs within 20
iterations. After the fix, they have survived > 500 iterations thus far.
I'll leave running overnight just to be sure.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1805256

Title:
  qemu-img hangs on rcu_call_ready_event logic in Aarch64 when
  converting images

Status in kunpeng920:
  Triaged
Status in kunpeng920 ubuntu-18.04 series:
  New
Status in kunpeng920 ubuntu-18.04-hwe series:
  New
Status in kunpeng920 ubuntu-19.10 series:
  New
Status in kunpeng920 ubuntu-20.04 series:
  New
Status in kunpeng920 upstream-kernel series:
  Fix Committed
Status in QEMU:
  In Progress
Status in qemu package in Ubuntu:
  Incomplete
Status in qemu source package in Bionic:
  Incomplete
Status in qemu source package in Disco:
  Incomplete
Status in qemu source package in Eoan:
  Incomplete
Status in qemu source package in Focal:
  Incomplete

Bug description:
  Command:

  qemu-img convert -f qcow2 -O qcow2 ./disk01.qcow2 ./output.qcow2

  Hangs indefinitely approximately 30% of the runs.

  

  Workaround:

  qemu-img convert -m 1 -f qcow2 -O qcow2 ./disk01.qcow2 ./output.qcow2

  Run "qemu-img convert" with "a single coroutine" to avoid this issue.

  

  (gdb) thread 1
  ...
  (gdb) bt
  #0 0xbf1ad81c in __GI_ppoll
  #1 0xaabcf73c in ppoll
  #2 qemu_poll_ns
  #3 0xaabd0764 in os_host_main_loop_wait
  #4 main_loop_wait
  ...

  (gdb) thread 2
  ...
  (gdb) bt
  #0 syscall ()
  #1 0xaabd41cc in qemu_futex_wait
  #2 qemu_event_wait (ev=ev@entry=0xaac86ce8 )
  #3 0xaabed05c in call_rcu_thread
  #4 0xaabd34c8 in qemu_thread_start
  #5 0xbf25c880 in start_thread
  #6 0xbf1b6b9c in thread_start ()

  (gdb) thread 3
  ...
  (gdb) bt
  #0 0xbf11aa20 in __GI___sigtimedwait
  #1 0xbf2671b4 in __sigwait
  #2 0xaabd1ddc in sigwait_compat
  #3 0xaabd34c8 in qemu_thread_start
  #4 0xbf25c880 in start_thread
  #5 0xbf1b6b9c in thread_start

  

  (gdb) run
  Starting program: /usr/bin/qemu-img convert -f qcow2 -O qcow2
  ./disk01.ext4.qcow2 ./output.qcow2

  [New Thread 0xbec5ad90 (LWP 72839)]
  [New Thread 0xbe459d90 (LWP 72840)]
  [New Thread 0xbdb57d90 (LWP 72841)]
  [New Thread 0xacac9d90 (LWP 72859)]
  [New Thread 0xa7ffed90 (LWP 72860)]
  [New Thread 0xa77fdd90 (LWP 72861)]
  [New Thread 0xa6ffcd90 (LWP 72862)]
  [New Thread 0xa67fbd90 (LWP 72863)]
  [New Thread 0xa5ffad90 (LWP 72864)]

  [Thread 0xa5ffad90 (LWP 72864) exited]
  [Thread 0xa6ffcd90 (LWP 72862) exited]
  [Thread 0xa77fdd90 (LWP 72861) exited]
  [Thread 0xbdb57d90 (LWP 72841) exited]
  [Thread 0xa67fbd90 (LWP 72863) exited]
  [Thread 0xacac9d90 (LWP 72859) exited]
  [Thread 0xa7ffed90 (LWP 72860) exited]

  
  """

  All the tasks left are blocked in a system call, so no task left to call
  qemu_futex_wake() to unblock thread #2 (in futex()), which would unblock
  thread #1 (doing poll() in a pipe with thread #2).

  Those 7 threads exit before disk conversion is complete (sometimes in
  the beginning, sometimes at the end).

  

  [ Original Description ]

  On the HiSilicon D06 system - a 96 core NUMA arm64 box - qemu-img
  frequently hangs (~50% of the time) with this command:

  qemu-img convert -f qcow2 -O qcow2 /tmp/cloudimg /tmp/cloudimg2

  Where "cloudimg" is a standard qcow2 Ubuntu cloud image. This
  qcow2->qcow2 conversion happens to be something uvtool does every time
  it fetches images.

  Once hung, attaching gdb gives the following backtrace:

  (gdb) bt
  #0  0xae4f8154 in __GI_ppoll (fds=0xe8a67dc0, 
nfds=187650274213760,
  timeout=, timeout@entry=0x0, sigmask=0xc123b950)
  at ../sysdeps/unix/sysv/linux/ppoll.c:39
  #1  0xbbefaf00 in ppoll (__ss=0x0, __timeout=0x0, __nfds=,
  __fds=) at /usr/include/aarch64-linux-gnu/bits/poll2.h:77
  #2  qemu_poll_ns (fds=, nfds=,
  timeout=timeout@entry=-1) at util/qemu-timer.c:322
  #3  0xbbefbf80 in os_host_main_loop_wait (timeout=-1)
  at util/main-loop.c:233
  #4  main_loop_wait (nonblocking=) at util/main-loop.c:497
  #5  0xbbe2aa30 in convert_do_copy (s=0xc123bb58) at 
qemu-img.c:1980
  #6  img_convert (argc=, argv=) at 
qemu-img.c:2456
  #7  0xbbe2333c in main (argc=7, argv=) at 
qemu-img.c:4975

  Reproduced w/ latest QEMU git (@ 53744e0a182)

To manage notifications about this bug go to:
https://bugs.launchpad.net/kunpeng920/+bug/1805256/+subscriptions

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag] patchew/20200505230941.3984108-1-alistair.fran...@wdc.com 
-> patchew/20200505230941.3984108-1-alistair.fran...@wdc.com
Switched to a new branch 'test'
258378d qom: Drop @errp parameter of object_property_del()
ae7803d spapr_pci: Drop some dead error handling
3867341 qdev: Unrealize must not fail
27ab964 Drop more @errp parameters after previous commit
051ab01 qom: Drop parameter @errp of object_property_add() & friends
d9987b4 qdev: Clean up qdev_connect_gpio_out_named()
4ac9492 hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
576bcc1 e1000: Don't run e1000_instance_init() twice
1e343fb hw/isa/superio: Make the components QOM children
2546372 s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
9feee4e tests/check-qom-proplist: Improve iterator coverage
e3f4e24 qom: Drop object_property_set_description() parameter @errp
2235c72 qom: Make all the object_property_add_FOO() return the property
4a961f6 qom: Drop convenience method object_property_get_uint16List()
a4ffec8 qom: Simplify object_property_get_enum()
30ca898 qom: Drop object_property_del_child()'s unused parameter @errp
77d819a qom: Clean up inconsistent use of gchar * vs. char *
96209f3 qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 96209f39f167 (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit 77d819acbc03 (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit 30ca898bb7dc (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit a4ffec863b8d (qom: Simplify object_property_get_enum())
5/18 Checking commit 4a961f67af22 (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit 2235c72d45a6 (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit e3f4e24ddb2e (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit 9feee4e3da40 (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit 2546372e6e85 (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit 1e343fbe99af (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit 576bcc126dca (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit 4ac949293617 (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit d9987b49f373 (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit 051ab01ed5fe (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit 27ab9642f5b9 (Drop more @errp parameters after previous 
commit)
16/18 Checking commit 386734149793 (qdev: Unrealize must not fail)
17/18 Checking commit ae7803d737f4 (spapr_pci: Drop some dead error handling)
18/18 Checking commit 258378dd975d (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[PULL 0/1] Register API Queue

[Alistair Francis]

The following changes since commit f19d118bed77bb95681b07f4e76dbb700c16918d:

  Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2020-05-04' into 
staging (2020-05-05 15:47:44 +0100)

are available in the Git repository at:

  g...@github.com:alistair23/qemu.git tags/pull-reg-to-apply-20200505

for you to fetch changes up to f08085f49fb66a5cdc86653bd896d0e728bcee50:

  hw/core/register: Add register_init_block8 helper. (2020-05-05 13:37:51 -0700)


Pull request for RegisterAPI

This is a single patch to add support to the RegisterAPI for different
data sizes.


Joaquin de Andres (1):
  hw/core/register: Add register_init_block8 helper.

 hw/core/register.c| 46 --
 include/hw/register.h |  8 
 2 files changed, 44 insertions(+), 10 deletions(-)

[PULL 1/1] hw/core/register: Add register_init_block8 helper.

[Alistair Francis]

From: Joaquin de Andres 

There was no support for 8 bits block registers. Changed
register_init_block32 to be generic and static, adding register
size in bits as parameter. Created one helper for each size.

Signed-off-by: Joaquin de Andres 
Message-Id: <20200402162839.76636-1...@xcancerberox.com.ar>
Reviewed-by: Alistair Francis 
Signed-off-by: Alistair Francis 
---
 hw/core/register.c| 46 +--
 include/hw/register.h |  8 
 2 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/hw/core/register.c b/hw/core/register.c
index 3c77396587..ddf91eb445 100644
--- a/hw/core/register.c
+++ b/hw/core/register.c
@@ -246,16 +246,18 @@ uint64_t register_read_memory(void *opaque, hwaddr addr,
 return extract64(read_val, 0, size * 8);
 }
 
-RegisterInfoArray *register_init_block32(DeviceState *owner,
- const RegisterAccessInfo *rae,
- int num, RegisterInfo *ri,
- uint32_t *data,
- const MemoryRegionOps *ops,
- bool debug_enabled,
- uint64_t memory_size)
+static RegisterInfoArray *register_init_block(DeviceState *owner,
+  const RegisterAccessInfo *rae,
+  int num, RegisterInfo *ri,
+  void *data,
+  const MemoryRegionOps *ops,
+  bool debug_enabled,
+  uint64_t memory_size,
+  size_t data_size_bits)
 {
 const char *device_prefix = object_get_typename(OBJECT(owner));
 RegisterInfoArray *r_array = g_new0(RegisterInfoArray, 1);
+int data_size = data_size_bits >> 3;
 int i;
 
 r_array->r = g_new0(RegisterInfo *, num);
@@ -264,12 +266,12 @@ RegisterInfoArray *register_init_block32(DeviceState 
*owner,
 r_array->prefix = device_prefix;
 
 for (i = 0; i < num; i++) {
-int index = rae[i].addr / 4;
+int index = rae[i].addr / data_size;
 RegisterInfo *r = [index];
 
 *r = (RegisterInfo) {
-.data = [index],
-.data_size = sizeof(uint32_t),
+.data = data + data_size * index,
+.data_size = data_size,
 .access = [i],
 .opaque = owner,
 };
@@ -284,6 +286,30 @@ RegisterInfoArray *register_init_block32(DeviceState 
*owner,
 return r_array;
 }
 
+RegisterInfoArray *register_init_block8(DeviceState *owner,
+const RegisterAccessInfo *rae,
+int num, RegisterInfo *ri,
+uint8_t *data,
+const MemoryRegionOps *ops,
+bool debug_enabled,
+uint64_t memory_size)
+{
+return register_init_block(owner, rae, num, ri, (void *)
+   data, ops, debug_enabled, memory_size, 8);
+}
+
+RegisterInfoArray *register_init_block32(DeviceState *owner,
+ const RegisterAccessInfo *rae,
+ int num, RegisterInfo *ri,
+ uint32_t *data,
+ const MemoryRegionOps *ops,
+ bool debug_enabled,
+ uint64_t memory_size)
+{
+return register_init_block(owner, rae, num, ri, (void *)
+   data, ops, debug_enabled, memory_size, 32);
+}
+
 void register_finalize_block(RegisterInfoArray *r_array)
 {
 object_unparent(OBJECT(_array->mem));
diff --git a/include/hw/register.h b/include/hw/register.h
index 5796584588..5d2c565ae0 100644
--- a/include/hw/register.h
+++ b/include/hw/register.h
@@ -185,6 +185,14 @@ uint64_t register_read_memory(void *opaque, hwaddr addr, 
unsigned size);
  *  memory region (r_array->mem) the caller should add to a container.
  */
 
+RegisterInfoArray *register_init_block8(DeviceState *owner,
+const RegisterAccessInfo *rae,
+int num, RegisterInfo *ri,
+uint8_t *data,
+const MemoryRegionOps *ops,
+bool debug_enabled,
+uint64_t memory_size);
+
 RegisterInfoArray *register_init_block32(DeviceState *owner,
  const RegisterAccessInfo *rae,
  int num, RegisterInfo *ri,

Re: [PATCH] riscv: Change the default behavior if no -bios option is specified

[Alistair Francis]

On Tue, May 5, 2020 at 1:34 PM Alistair Francis  wrote:
>
> On Fri, May 1, 2020 at 5:21 AM Bin Meng  wrote:
> >
> > From: Bin Meng 
> >
> > Per QEMU deprecated doc, QEMU 4.1 introduced support for the -bios
> > option in QEMU for RISC-V for the virt machine and sifive_u machine.
> > The default behavior has been that QEMU does not automatically load
> > any firmware if no -bios option is included.
> >
> > Now 2 releases passed, it's time to change the default behavior to
> > load the default OpenSBI firmware automatically. The firmware is
> > included with the QEMU release and no user interaction is required.
> > All a user needs to do is specify the kernel they want to boot with
> > the -kernel option.
> >
> > Signed-off-by: Bin Meng 
>
> Thanks!
>
> Reviewed-by: Alistair Francis 
>
> Applied to the RISC-V tree.

This fails `make check`

qemu-system-riscv64: Unable to load the RISC-V firmware
"opensbi-riscv64-spike-fw_jump.elf"
Broken pipe
/scratch/alistair/software/qemu/tests/qtest/libqtest.c:166:
kill_qemu() tried to terminate QEMU process but encountered exit
status 1 (expected 0)
ERROR - too few tests run (expected 7, got 2)
make: *** [/scratch/alistair/software/qemu/tests/Makefile.include:637:
check-qtest-riscv64] Error 1

Alistair

>
> Alistair
>
> > ---
> >
> >  hw/riscv/boot.c | 31 ---
> >  1 file changed, 4 insertions(+), 27 deletions(-)
> >
> > diff --git a/hw/riscv/boot.c b/hw/riscv/boot.c
> > index 726300a..b76b2f3 100644
> > --- a/hw/riscv/boot.c
> > +++ b/hw/riscv/boot.c
> > @@ -41,34 +41,11 @@ void riscv_find_and_load_firmware(MachineState *machine,
> >  {
> >  char *firmware_filename = NULL;
> >
> > -if (!machine->firmware) {
> > +if ((!machine->firmware) || (!strcmp(machine->firmware, "default"))) {
> >  /*
> > - * The user didn't specify -bios.
> > - * At the moment we default to loading nothing when this hapens.
> > - * In the future this defaul will change to loading the prebuilt
> > - * OpenSBI firmware. Let's warn the user and then continue.
> > -*/
> > -if (!qtest_enabled()) {
> > -warn_report("No -bios option specified. Not loading a 
> > firmware.");
> > -warn_report("This default will change in a future QEMU 
> > release. " \
> > -"Please use the -bios option to avoid breakages 
> > when "\
> > -"this happens.");
> > -warn_report("See QEMU's deprecation documentation for 
> > details.");
> > -}
> > -return;
> > -}
> > -
> > -if (!strcmp(machine->firmware, "default")) {
> > -/*
> > - * The user has specified "-bios default". That means we are going 
> > to
> > - * load the OpenSBI binary included in the QEMU source.
> > - *
> > - * We can't load the binary by default as it will break existing 
> > users
> > - * as users are already loading their own firmware.
> > - *
> > - * Let's try to get everyone to specify the -bios option at all 
> > times,
> > - * so then in the future we can make "-bios default" the default 
> > option
> > - * if no -bios option is set without breaking anything.
> > + * The user didn't specify -bios, or has specified "-bios default".
> > + * That means we are going to load the OpenSBI binary included in
> > + * the QEMU source.
> >   */
> >  firmware_filename = riscv_find_firmware(default_machine_firmware);
> >  } else if (strcmp(machine->firmware, "none")) {
> > --
> > 2.7.4
> >
> >

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Hi Seiger,
I am not an expert on libvirt. I mostly use qemu command line for my test. I 
was able to achieve the 3960X configuration with the following command line. 

# qemu-system-x86_64 -name rhel7  -m 16384 -smp
24,cores=12,threads=2,sockets=1 -hda vdisk.qcow2 -enable-kvm -net nic
-net bridge,br=virbr0,helper=/usr/libexec/qemu-bridge-helper -cpu
host,+topoext -nographic -numa node,nodeid=0,cpus=0-5 -numa
node,nodeid=1,cpus=6-11 -numa node,nodeid=2,cpus=12-17 -numa
node,nodeid=3,cpus=18-23

Basically qemu does not have all the information to build the topology
for every configuration. It depends on  libvirt for that information.
See if this combination works for you.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    
  EPYC-IBPB
  AMD
  

  In windows, coreinfo reports correctly:

    Unified Cache 1, Level 3,8 MB, Assoc  16, LineSize  64
    Unified Cache 6, Level 3,8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   
  EPYC-IBPB
  AMD
  

  in windows, coreinfo reports incorrectly:

  --  Unified Cache  1, Level 3,8 MB, Assoc  16, LineSize  64
  **  Unified Cache  6, Level 3,8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies 
parameter:
   

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

Re: [PATCH v5 19/31] qcow2: Add subcluster support to calculate_l2_meta()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

If an image has subclusters then there are more copy-on-write
scenarios that we need to consider. Let's say we have a write request
from the middle of subcluster #3 until the end of the cluster:

1) If we are writing to a newly allocated cluster then we need
copy-on-write. The previous contents of subclusters #0 to #3 must
be copied to the new cluster. We can optimize this process by
skipping all leading unallocated or zero subclusters (the status of
those skipped subclusters will be reflected in the new L2 bitmap).

2) If we are overwriting an existing cluster:

2.1) If subcluster #3 is unallocated or has the all-zeroes bit set
 then we need copy-on-write (on subcluster #3 only).

2.2) If subcluster #3 was already allocated then there is no need
 for any copy-on-write. However we still need to update the L2
 bitmap to reflect possible changes in the allocation status of
 subclusters #4 to #31. Because of this, this function checks
 if all the overwritten subclusters are already allocated and
 in this case it returns without creating a new QCowL2Meta
 structure.

After all these changes l2meta_cow_start() and l2meta_cow_end()
are not necessarily cluster-aligned anymore. We need to update the
calculation of old_start and old_end in handle_dependencies() to
guarantee that no two requests try to write on the same cluster.

Signed-off-by: Alberto Garcia 
---
  block/qcow2-cluster.c | 174 +++---
  1 file changed, 146 insertions(+), 28 deletions(-)




-/* Return if there's no COW (all clusters are normal and we keep them) */
+/* Return if there's no COW (all subclusters are normal and we are
+ * keeping the clusters) */
  if (keep_old) {
+unsigned first_sc = cow_start_to / s->subcluster_size;
+unsigned last_sc = (cow_end_from - 1) / s->subcluster_size;
  int i;
-for (i = 0; i < nb_clusters; i++) {
-l2_entry = get_l2_entry(s, l2_slice, l2_index + i);
-if (qcow2_get_cluster_type(bs, l2_entry) != QCOW2_CLUSTER_NORMAL) {
+for (i = first_sc; i <= last_sc; i++) {
+unsigned c = i / s->subclusters_per_cluster;
+unsigned sc = i % s->subclusters_per_cluster;
+l2_entry = get_l2_entry(s, l2_slice, l2_index + c);
+l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index + c);
+type = qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, sc);
+if (type == QCOW2_SUBCLUSTER_INVALID) {
+l2_index += c; /* Point to the invalid entry */
+goto fail;
+}
+if (type != QCOW2_SUBCLUSTER_NORMAL) {
  break;
  }
  }


This loop is now 32 times slower (for extended L2 entries).  Do you 
really need to check for an invalid subcluster here, or can we just 
blindly check that all 32 subclusters are NORMAL, and leave handling of 
invalid clusters for the rest of the function after we failed the 
exit-early test?  For that matter, for all but the first and last 
cluster, checking if 32 clusters are NORMAL is a simple 64-bit 
comparison rather than 32 iterations of a loop; and even for the first 
and last cluster, the _RANGE macros in 14/31 work well to mask out which 
bits must be set/cleared.  My guess is that optimizing this loop is 
worthwhile, since overwriting existing data is probably more common than 
allocating new data.



-if (i == nb_clusters) {
-return;
+if (i == last_sc + 1) {
+return 0;
  }
  }


If we get here, then i is now the address of the first subcluster that 
was not NORMAL, even if it is much smaller than the final subcluster 
learned by nb_clusters for the overall request.  [1]


  
  /* Get the L2 entry of the first cluster */

  l2_entry = get_l2_entry(s, l2_slice, l2_index);
-type = qcow2_get_cluster_type(bs, l2_entry);
+l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index);
+sc_index = offset_to_sc_index(s, guest_offset);
+type = qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, sc_index);
  
-if (type == QCOW2_CLUSTER_NORMAL && keep_old) {

-cow_start_from = cow_start_to;
+if (type == QCOW2_SUBCLUSTER_INVALID) {
+goto fail;
+}
+
+if (!keep_old) {
+switch (type) {
+case QCOW2_SUBCLUSTER_COMPRESSED:
+cow_start_from = 0;
+break;
+case QCOW2_SUBCLUSTER_NORMAL:
+case QCOW2_SUBCLUSTER_ZERO_ALLOC:
+case QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC: {
+int i;
+/* Skip all leading zero and unallocated subclusters */
+for (i = 0; i < sc_index; i++) {
+QCow2SubclusterType t;
+t = qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, i);
+if (t == QCOW2_SUBCLUSTER_INVALID) {
+goto fail;
+  

linux-user - time64 question

[Sid Manning]

I’m looking at a testcase failure when my target uses 64bit time in msg.h 
(struct msqid_ds).  I’ve been able to get around this but changing 
target_msqid_ds like so:

@@ -3900,18 +3901,9 @@ static inline abi_long do_semop(int semid, abi_long ptr,
unsigned nsops)
 struct target_msqid_ds
 {
 struct target_ipc_perm msg_perm;
-abi_ulong msg_stime;
-#if TARGET_ABI_BITS == 32
-abi_ulong __unused1;
-#endif
-abi_ulong msg_rtime;
-#if TARGET_ABI_BITS == 32
-abi_ulong __unused2;
-#endif
-abi_ulong msg_ctime;
-#if TARGET_ABI_BITS == 32
-abi_ulong __unused3;
-#endif
+abi_ullong msg_stime;
+abi_ullong msg_rtime;
+abi_ullong msg_ctime;
 abi_ulong __msg_cbytes;
 abi_ulong msg_qnum;
 abi_ulong msg_qbytes;

It seems like either should have worked but I get garbage back in some of the 
elements below msg_time fields without the change.

If time_t is 64bits then it seems like stime/rtime/ctime should be abi_ullong.

My target is Hexagon and the TARGET_ABI_BITS is 32.

Thanks,

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
beb6c50 qom: Drop @errp parameter of object_property_del()
d0d55e6 spapr_pci: Drop some dead error handling
ec2df94 qdev: Unrealize must not fail
5b8de9e Drop more @errp parameters after previous commit
edf94c6 qom: Drop parameter @errp of object_property_add() & friends
6ff8f7d qdev: Clean up qdev_connect_gpio_out_named()
46da334 hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
6a5bc55 e1000: Don't run e1000_instance_init() twice
c4844bd hw/isa/superio: Make the components QOM children
7be19fd s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
d09d20e tests/check-qom-proplist: Improve iterator coverage
548a3c9 qom: Drop object_property_set_description() parameter @errp
ac198ff qom: Make all the object_property_add_FOO() return the property
66c5d26 qom: Drop convenience method object_property_get_uint16List()
4853b5b qom: Simplify object_property_get_enum()
4f9c21f qom: Drop object_property_del_child()'s unused parameter @errp
b9913e2 qom: Clean up inconsistent use of gchar * vs. char *
050e01e qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 050e01e9c03d (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit b9913e2e09ab (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit 4f9c21f8e769 (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit 4853b5ba881d (qom: Simplify object_property_get_enum())
5/18 Checking commit 66c5d264f12d (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit ac198ffe9e7a (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit 548a3c97f4bf (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit d09d20e67177 (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit 7be19fd93345 (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit c4844bd223f1 (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit 6a5bc55a6703 (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit 46da33429c8f (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit 6ff8f7d6be67 (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit edf94c680754 (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit 5b8de9ee567a (Drop more @errp parameters after previous 
commit)
16/18 Checking commit ec2df94015ef (qdev: Unrealize must not fail)
17/18 Checking commit d0d55e6ba917 (spapr_pci: Drop some dead error handling)
18/18 Checking commit beb6c509cd49 (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [PATCH v2 6/6] iotests: Add test 291 to for qemu-img bitmap coverage

[Eric Blake]

On 5/4/20 8:05 AM, Max Reitz wrote:

On 21.04.20 23:20, Eric Blake wrote:

Add a new test covering the 'qemu-img bitmap' subcommand, as well as
'qemu-img convert --bitmaps', both added in recent patches.

Signed-off-by: Eric Blake 



+echo
+echo "=== Bitmap preservation not possible to non-qcow2 ==="
+echo
+
+mv "$TEST_IMG" "$TEST_IMG.orig"


“mv” doesn’t work images with external data files.

(ORIG_IMG=$TEST_IMG; TEST_IMG="$TEST_IMG".orig should work)


Good idea.




+$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG"
+
+echo
+echo "=== Convert with bitmap preservation ==="
+echo
+
+# Only bitmaps from the active layer are copied


That’s kind of obvious when you think about (whenever an image is
attached to a VM, only the active layer’s bitmaps are visible, not those
from the backing chain), but maybe this should be noted in the
documentation?


As part of integrating bitmaps with external snapshots, libvirt actually 
depends on being able to see bitmaps from the backing chain - but as 
bitmaps are always referenced as a 'node name, bitmap name' tuple, this 
is indeed doable.





+$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG"
+$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+# But we can also merge in bitmaps from other layers
+$QEMU_IMG bitmap --add --disabled -f $IMGFMT "$TEST_IMG" b0
+$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG" tmp
+$QEMU_IMG bitmap --merge b0 -b "$TEST_IMG.base" -F $IMGFMT "$TEST_IMG" tmp
+$QEMU_IMG bitmap --merge tmp "$TEST_IMG" b0
+$QEMU_IMG bitmap --remove -f $IMGFMT "$TEST_IMG" tmp


Why do we need tmp here?  Can’t we just merge base’s b0 directly into
$TEST_IMG’s b0?


Yes, we could.  But then I wouldn't cover as many bitmap subcommands. 
Adding a comment about why the example is contrived (for maximal 
coverage) is a good idea.




+=== Check bitmap contents ===
+
+[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET},
+{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": 
false},
+{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET}]
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET},
+{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": 
false},
+{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET}]
+[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET},
+{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": 
false},
+{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, 
"offset": OFFSET}]


Am I looking at this wrong or does the bitmap data seem to be inverted?
  Everywhere where I’d expect the bitmaps to be cleared, this map reports
data=true, whereas where I’d expect them to be set, it reports data=false.

I suppose that’s intentional, but can you explain this behavior to me?


This is an artifact of how x-dirty-bitmap works (it has the x- prefix 
because it is a hack, but we don't have anything better for reading out 
bitmap contents).  The NBD spec returns block status as a 32-bit value 
for a 'metadata context'; normally, we use context 'base:allocation' 
context where bit 0 is set for holes or clear for allocated, and bit 1 
is set for reads-as-zero or clear for unknown contents (favoring all-0 
as the most-common case).  But with x-dirty-bitmap, we are instead 
abusing NBD to query the 'qemu:dirty-bitmap:FOO' context, where bit 0 is 
set for anywhere the bitmap has a 1, yet feed that information into the 
pre-existing qemu code for handling block status.  So qemu-img map is 
reporting "data":true for what it thinks is the normal 0-for-allocated, 
and "data":false for 1-for-sparse, and we just have to translate that 
back into an understanding of what the bitmap reported.  Yes, a comment 
would be helpful.


I would _really_ love to enhance 'qemu-img map' to output image-specific 
metadata _in addition_ to the existing "zero" and "data" fields (by 
having qemu-img read two NBD contexts at once: both base:allocation and 
qemu:dirty-bitmap:FOO), at which point we can drop the x- prefix and 
avoid the abuse of qemu's internals by overwriting the block_status 
code.  But that's a bigger project.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v5 15/31] qcow2: Add qcow2_cluster_is_allocated()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

This helper function tells us if a cluster is allocated (that is,
there is an associated host offset for it).

Signed-off-by: Alberto Garcia 
---
  block/qcow2.h | 6 ++
  1 file changed, 6 insertions(+)


Reviewed-by: Eric Blake 



diff --git a/block/qcow2.h b/block/qcow2.h
index be7816a3b8..b5db8d2f36 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -763,6 +763,12 @@ QCow2SubclusterType 
qcow2_get_subcluster_type(BlockDriverState *bs,
  }
  }
  
+static inline bool qcow2_cluster_is_allocated(QCow2ClusterType type)

+{
+return (type == QCOW2_CLUSTER_COMPRESSED || type == QCOW2_CLUSTER_NORMAL ||
+type == QCOW2_CLUSTER_ZERO_ALLOC);
+}
+
  /* Check whether refcounts are eager or lazy */
  static inline bool qcow2_need_accurate_refcounts(BDRVQcow2State *s)
  {



--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v5 14/31] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

This patch adds QCow2SubclusterType, which is the subcluster-level
version of QCow2ClusterType. All QCOW2_SUBCLUSTER_* values have the
the same meaning as their QCOW2_CLUSTER_* equivalents (when they
exist). See below for details and caveats.

In images without extended L2 entries clusters are treated as having
exactly one subcluster so it is possible to replace one data type with
the other while keeping the exact same semantics.

With extended L2 entries there are new possible values, and every
subcluster in the same cluster can obviously have a different
QCow2SubclusterType so functions need to be adapted to work on the
subcluster level.

There are several things that have to be taken into account:

   a) QCOW2_SUBCLUSTER_COMPRESSED means that the whole cluster is
  compressed. We do not support compression at the subcluster
  level.

   b) There are two different values for unallocated subclusters:
  QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN which means that the whole
  cluster is unallocated, and QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  which means that the cluster is allocated but the subcluster is
  not. The latter can only happen in images with extended L2
  entries.


Or put differently, extents of the qcow2 file are always allocated a 
contiguous cluster at a time (so using larger clusters reduces 
fragmentation), but because we can now defer to the backing image a 
sub-cluster at a time, we have less I/O to perform the first time the 
guest touches a subcluster.  The two different return values thus tell 
us when we need to do a cluster allocation vs. just an in-place 
overwrite or a sub-cluster COW.




   c) QCOW2_SUBCLUSTER_INVALID is used to detect the cases where an L2
  entry has a value that violates the specification. The caller is
  responsible for handling these situations.

  To prevent compatibility problems with images that have invalid
  values but are currently being read by QEMU without causing side
  effects, QCOW2_SUBCLUSTER_INVALID is only returned for images
  with extended L2 entries.

qcow2_cluster_to_subcluster_type() is added as a separate function
from qcow2_get_subcluster_type(), but this is only temporary and both
will be merged in a subsequent patch.

Signed-off-by: Alberto Garcia 
---
  block/qcow2.h | 127 +-
  1 file changed, 126 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 4ad93772b9..be7816a3b8 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -80,6 +80,21 @@
  
  #define QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER 32
  
+/* The subcluster X [0..31] is allocated */

+#define QCOW_OFLAG_SUB_ALLOC(X)   (1ULL << (X))
+/* The subcluster X [0..31] reads as zeroes */
+#define QCOW_OFLAG_SUB_ZERO(X)(QCOW_OFLAG_SUB_ALLOC(X) << 32)
+/* Subclusters X to Y (both included) are allocated */
+#define QCOW_OFLAG_SUB_ALLOC_RANGE(X, Y) \
+(QCOW_OFLAG_SUB_ALLOC((Y) + 1) - QCOW_OFLAG_SUB_ALLOC(X))


Nicer than my initial thoughts on getting rid of the bit-wise loop.  And 
uses 64-bit math to produce a 32-bit answer, so there are no edge cases 
where overflow could misbehave even though the intermediate steps may 
require 33 bits.  Works as long as X <= Y (should that be mentioned in 
the contract?)



+/* Subclusters X to Y (both included) read as zeroes */
+#define QCOW_OFLAG_SUB_ZERO_RANGE(X, Y) \
+(QCOW_OFLAG_SUB_ALLOC_RANGE(X, Y) << 32)


Also works (you do the math in the low 33 bits before shifting), again 
if X <= Y.



+/* L2 entry bitmap with all allocation bits set */
+#define QCOW_L2_BITMAP_ALL_ALLOC  (QCOW_OFLAG_SUB_ALLOC_RANGE(0, 31))
+/* L2 entry bitmap with all "read as zeroes" bits set */
+#define QCOW_L2_BITMAP_ALL_ZEROES (QCOW_OFLAG_SUB_ZERO_RANGE(0, 31))


More complicated than merely writing 0xULL and 
(0xULL<<32), but the compiler will constant-fold it to the same 
value, and it elegantly expresses the intent.  I like it.


Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v4 00/14] LUKS: encryption slot management using amend interface

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505200819.5662-1-mlevi...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505200819.5662-1-mlevi...@redhat.com
Subject: [PATCH v4 00/14] LUKS: encryption slot management using amend interface
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
a66c1e0 iotests: add tests for blockdev-amend
d8564bf block/qcow2: implement blockdev-amend
572ee9d block/crypto: implement blockdev-amend
5f988ac block/core: add generic infrastructure for x-blockdev-amend qmp command
92daa6a iotests: qemu-img tests for luks key management
621c9e3 iotests: filter few more luks specific create options
d465502 block/qcow2: extend qemu-img amend interface with crypto options
86e6207 block/crypto: implement the encryption key management
0f8bfce block/crypto: rename two functions
85542a6 block/amend: refactor qcow2 amend options
f0f6efb block/amend: separate amend and create options for qemu-img
771c331 block/amend: add 'force' option
29c7c9c qcrypto/luks: implement encryption key management
97f7f25 qcrypto/core: add generic infrastructure for crypto options amendment

=== OUTPUT BEGIN ===
1/14 Checking commit 97f7f254b229 (qcrypto/core: add generic infrastructure for 
crypto options amendment)
2/14 Checking commit 29c7c9cc9ace (qcrypto/luks: implement encryption key 
management)
3/14 Checking commit 771c331f6773 (block/amend: add 'force' option)
4/14 Checking commit f0f6efbb1ed3 (block/amend: separate amend and create 
options for qemu-img)
ERROR: Macros with multiple statements should be enclosed in a do - while loop
#31: FILE: block/qcow2.c:5498:
+#define QCOW_COMMON_OPTIONS \
+{   \
+.name = BLOCK_OPT_SIZE, \
+.type = QEMU_OPT_SIZE,  \
+.help = "Virtual disk size" \
+},  \
+{   \
+.name = BLOCK_OPT_COMPAT_LEVEL, \
+.type = QEMU_OPT_STRING,\
+.help = "Compatibility level (v2 [0.10] or v3 [1.1])"   \
+},  \
+{   \
+.name = BLOCK_OPT_BACKING_FILE, \
+.type = QEMU_OPT_STRING,\
+.help = "File name of a base image" \
+},  \
+{   \
+.name = BLOCK_OPT_BACKING_FMT,  \
+.type = QEMU_OPT_STRING,\
+.help = "Image format of the base image"\
+},  \
+{   \
+.name = BLOCK_OPT_DATA_FILE,\
+.type = QEMU_OPT_STRING,\
+.help = "File name of an external data file"\
+},  \
+{   \
+.name = BLOCK_OPT_DATA_FILE_RAW,\
+.type = QEMU_OPT_BOOL,  \
+.help = "The external data file must stay valid "   \
+"as a raw image"\
+},  \
+{   \
+.name = BLOCK_OPT_ENCRYPT,  \
+.type = QEMU_OPT_BOOL,  \
+.help = "Encrypt the image with format 'aes'. (Deprecated " \
+"in favor of " BLOCK_OPT_ENCRYPT_FORMAT "=aes)",\
+},  \
+{   \
+.name = BLOCK_OPT_ENCRYPT_FORMAT,   \
+.type = QEMU_OPT_STRING,\
+.help = "Encrypt the image, format choices: 'aes', 'luks'", \
+},  \
+

Re: [PATCH v5 07/18] nvme: add max_ioqpairs device parameter

[Klaus Jensen]

On May  5 07:48, Klaus Jensen wrote:
> From: Klaus Jensen 
> 
> The num_queues device paramater has a slightly confusing meaning because
> it accounts for the admin queue pair which is not really optional.
> Secondly, it is really a maximum value of queues allowed.
> 
> Add a new max_ioqpairs parameter that only accounts for I/O queue pairs,
> but keep num_queues for compatibility.
> 
> Signed-off-by: Klaus Jensen 
> Reviewed-by: Maxim Levitsky 
> Reviewed-by: Philippe Mathieu-Daudé 
> Reviewed-by: Keith Busch 
> ---
>  hw/block/nvme.c | 51 ++---
>  hw/block/nvme.h |  3 ++-
>  2 files changed, 33 insertions(+), 21 deletions(-)
> 
> diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> index 623a88be93dc..3875a5f3dcbf 100644
> --- a/hw/block/nvme.c
> +++ b/hw/block/nvme.c
> @@ -1571,7 +1581,8 @@ static Property nvme_props[] = {
>   HostMemoryBackend *),
>  DEFINE_PROP_STRING("serial", NvmeCtrl, params.serial),
>  DEFINE_PROP_UINT32("cmb_size_mb", NvmeCtrl, params.cmb_size_mb, 0),
> -DEFINE_PROP_UINT32("num_queues", NvmeCtrl, params.num_queues, 64),
> +DEFINE_PROP_UINT32("num_queues", NvmeCtrl, params.num_queues, 0),
> +DEFINE_PROP_UINT32("max_ioqpairs", NvmeCtrl, params.max_ioqpairs, 64),
>  DEFINE_PROP_END_OF_LIST(),
>  };
>  

I noticed that this default of 64 makes the default configuration
unsafe by allowing the cq->cqid < 64 assert in nvme_irq_{,de}assert() to
trigger if the pin-based interrupt logic is used (under SPDK for
instance). The assert protects against undefined behavior caused by
shifting by more than 63 into the 64 bit irq_status variable.

As far as I can tell, the assert, the shift and the size of the
irq_status variable is bogus, so I posted a patch for this in
"hw/block/nvme: fixes for interrupt behavior". Preferably that should go
in before this series.

Re: [PATCH v2] target/riscv: fix check of guest pa top bits

[Alistair Francis]

On Fri, May 1, 2020 at 11:51 AM Jose Martins  wrote:
>
> The spec states that on sv39x4 guest physical  "address bits 63:41 must
> all be zeros, or else a guest-page-fault exception occurs.".  However,
> the check performed for these top bits of the virtual address on the
> second stage is the same as the one performed for virtual addresses on
> the first stage except with the 2-bit extension, effectively creating
> the same kind of "hole" in the guest's physical address space. I believe
> the following patch fixes this issue:
>
> Signed-off-by: Jose Martins 

Reviewed-by: Alistair Francis 

Applied to RISC-V tree.

Alistair

> ---
>  target/riscv/cpu_helper.c | 20 +---
>  1 file changed, 13 insertions(+), 7 deletions(-)
>
> diff --git a/target/riscv/cpu_helper.c b/target/riscv/cpu_helper.c
> index 247304d850..ae22c30bdd 100644
> --- a/target/riscv/cpu_helper.c
> +++ b/target/riscv/cpu_helper.c
> @@ -426,15 +426,21 @@ static int get_physical_address(CPURISCVState *env, 
> hwaddr *physical,
>  int va_bits = PGSHIFT + levels * ptidxbits + widened;
>  target_ulong mask, masked_msbs;
>
> -if (TARGET_LONG_BITS > (va_bits - 1)) {
> -mask = (1L << (TARGET_LONG_BITS - (va_bits - 1))) - 1;
> +if (!first_stage) {
> +if ((addr >> va_bits) != 0) {
> +return TRANSLATE_FAIL;
> +}
>  } else {
> -mask = 0;
> -}
> -masked_msbs = (addr >> (va_bits - 1)) & mask;
> +if (TARGET_LONG_BITS > (va_bits - 1)) {
> +mask = (1L << (TARGET_LONG_BITS - (va_bits - 1))) - 1;
> +} else {
> +mask = 0;
> +}
> +masked_msbs = (addr >> (va_bits - 1)) & mask;
>
> -if (masked_msbs != 0 && masked_msbs != mask) {
> -return TRANSLATE_FAIL;
> +if (masked_msbs != 0 && masked_msbs != mask) {
> +return TRANSLATE_FAIL;
> +}
>  }
>
>  int ptshift = (levels - 1) * ptidxbits;
> --
> 2.25.1
>
>

Re: [PATCH] riscv: Change the default behavior if no -bios option is specified

[Alistair Francis]

On Fri, May 1, 2020 at 5:21 AM Bin Meng  wrote:
>
> From: Bin Meng 
>
> Per QEMU deprecated doc, QEMU 4.1 introduced support for the -bios
> option in QEMU for RISC-V for the virt machine and sifive_u machine.
> The default behavior has been that QEMU does not automatically load
> any firmware if no -bios option is included.
>
> Now 2 releases passed, it's time to change the default behavior to
> load the default OpenSBI firmware automatically. The firmware is
> included with the QEMU release and no user interaction is required.
> All a user needs to do is specify the kernel they want to boot with
> the -kernel option.
>
> Signed-off-by: Bin Meng 

Thanks!

Reviewed-by: Alistair Francis 

Applied to the RISC-V tree.

Alistair

> ---
>
>  hw/riscv/boot.c | 31 ---
>  1 file changed, 4 insertions(+), 27 deletions(-)
>
> diff --git a/hw/riscv/boot.c b/hw/riscv/boot.c
> index 726300a..b76b2f3 100644
> --- a/hw/riscv/boot.c
> +++ b/hw/riscv/boot.c
> @@ -41,34 +41,11 @@ void riscv_find_and_load_firmware(MachineState *machine,
>  {
>  char *firmware_filename = NULL;
>
> -if (!machine->firmware) {
> +if ((!machine->firmware) || (!strcmp(machine->firmware, "default"))) {
>  /*
> - * The user didn't specify -bios.
> - * At the moment we default to loading nothing when this hapens.
> - * In the future this defaul will change to loading the prebuilt
> - * OpenSBI firmware. Let's warn the user and then continue.
> -*/
> -if (!qtest_enabled()) {
> -warn_report("No -bios option specified. Not loading a 
> firmware.");
> -warn_report("This default will change in a future QEMU release. 
> " \
> -"Please use the -bios option to avoid breakages when 
> "\
> -"this happens.");
> -warn_report("See QEMU's deprecation documentation for details.");
> -}
> -return;
> -}
> -
> -if (!strcmp(machine->firmware, "default")) {
> -/*
> - * The user has specified "-bios default". That means we are going to
> - * load the OpenSBI binary included in the QEMU source.
> - *
> - * We can't load the binary by default as it will break existing 
> users
> - * as users are already loading their own firmware.
> - *
> - * Let's try to get everyone to specify the -bios option at all 
> times,
> - * so then in the future we can make "-bios default" the default 
> option
> - * if no -bios option is set without breaking anything.
> + * The user didn't specify -bios, or has specified "-bios default".
> + * That means we are going to load the OpenSBI binary included in
> + * the QEMU source.
>   */
>  firmware_filename = riscv_find_firmware(default_machine_firmware);
>  } else if (strcmp(machine->firmware, "none")) {
> --
> 2.7.4
>
>

[PATCH 1/2] hw/block/nvme: fix pin-based interrupt behavior

[Klaus Jensen]

From: Klaus Jensen 

First, since the device only supports MSI-X or pin-based interrupt, if
MSI-X is not enabled, it should not accept interrupt vectors different
from 0 when creating completion queues.

Secondly, the irq_status NvmeCtrl member is meant to be compared to the
INTMS register, so it should only be 32 bits wide. And it is really only
useful when used with multi-message MSI.

Third, since we do not force a 1-to-1 correspondence between cqid and
interrupt vector, the irq_status register should not have bits set
according to cqid, but according to the associated interrupt vector.

Fix these issues, but keep irq_status available so we can easily support
multi-message MSI down the line.

Fixes: 5e9aa92eb1a5 ("hw/block: Fix pin-based interrupt behaviour of NVMe")
Signed-off-by: Klaus Jensen 
---
 hw/block/nvme.c | 12 
 hw/block/nvme.h |  2 +-
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 9b453423cf2c..2b072bbb21e7 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -127,8 +127,8 @@ static void nvme_irq_assert(NvmeCtrl *n, NvmeCQueue *cq)
 msix_notify(&(n->parent_obj), cq->vector);
 } else {
 trace_nvme_irq_pin();
-assert(cq->cqid < 64);
-n->irq_status |= 1 << cq->cqid;
+assert(cq->vector < 32);
+n->irq_status |= 1 << cq->vector;
 nvme_irq_check(n);
 }
 } else {
@@ -142,8 +142,8 @@ static void nvme_irq_deassert(NvmeCtrl *n, NvmeCQueue *cq)
 if (msix_enabled(&(n->parent_obj))) {
 return;
 } else {
-assert(cq->cqid < 64);
-n->irq_status &= ~(1 << cq->cqid);
+assert(cq->vector < 32);
+n->irq_status &= ~(1 << cq->vector);
 nvme_irq_check(n);
 }
 }
@@ -642,6 +642,10 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
 trace_nvme_err_invalid_create_cq_addr(prp1);
 return NVME_INVALID_FIELD | NVME_DNR;
 }
+if (unlikely(!msix_enabled(>parent_obj) && vector)) {
+trace_nvme_err_invalid_create_cq_vector(vector);
+return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
+}
 if (unlikely(vector > n->num_queues)) {
 trace_nvme_err_invalid_create_cq_vector(vector);
 return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 6520a9f0bead..db62589247d7 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -78,7 +78,7 @@ typedef struct NvmeCtrl {
 uint32_tcmbsz;
 uint32_tcmbloc;
 uint8_t *cmbuf;
-uint64_tirq_status;
+uint32_tirq_status;
 uint64_thost_timestamp; /* Timestamp sent by the host 
*/
 uint64_ttimestamp_set_qemu_clock_ms;/* QEMU clock time */
 
-- 
2.26.2

[PATCH 0/2] hw/block/nvme: fixes for interrupt behavior

[Klaus Jensen]

From: Klaus Jensen 

Klaus Jensen (2):
  hw/block/nvme: fix pin-based interrupt behavior
  hw/block/nvme: allow use of any valid msix vector

 hw/block/nvme.c | 14 +-
 hw/block/nvme.h |  2 +-
 2 files changed, 10 insertions(+), 6 deletions(-)

-- 
2.26.2

[PATCH 2/2] hw/block/nvme: allow use of any valid msix vector

[Klaus Jensen]

From: Klaus Jensen 

If the device uses MSI-X, any of the 2048 MSI-X interrupt vectors are
valid. If the device is not using MSI-X, vector will and can only be
zero at this point.

Signed-off-by: Klaus Jensen 
---
 hw/block/nvme.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 2b072bbb21e7..755ced8b03fb 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -646,7 +646,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
 trace_nvme_err_invalid_create_cq_vector(vector);
 return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
 }
-if (unlikely(vector > n->num_queues)) {
+if (unlikely(vector > PCI_MSIX_FLAGS_QSIZE)) {
 trace_nvme_err_invalid_create_cq_vector(vector);
 return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
 }
-- 
2.26.2

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
b6bd21f qom: Drop @errp parameter of object_property_del()
f308ad4 spapr_pci: Drop some dead error handling
fdb5400 qdev: Unrealize must not fail
7345bac Drop more @errp parameters after previous commit
2f039db qom: Drop parameter @errp of object_property_add() & friends
08871f2 qdev: Clean up qdev_connect_gpio_out_named()
def7ba0 hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
6fd6ae5 e1000: Don't run e1000_instance_init() twice
ff6a512 hw/isa/superio: Make the components QOM children
93b2971 s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
4e28c88 tests/check-qom-proplist: Improve iterator coverage
bd33afa qom: Drop object_property_set_description() parameter @errp
b6330ad qom: Make all the object_property_add_FOO() return the property
86270ef qom: Drop convenience method object_property_get_uint16List()
3293742 qom: Simplify object_property_get_enum()
1b625f3 qom: Drop object_property_del_child()'s unused parameter @errp
3fd6f46 qom: Clean up inconsistent use of gchar * vs. char *
37691e1 qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 37691e1e2c1b (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit 3fd6f465109b (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit 1b625f3ddbbc (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit 32937422d926 (qom: Simplify object_property_get_enum())
5/18 Checking commit 86270ef5b2fa (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit b6330ada2d04 (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit bd33afaebb53 (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit 4e28c881db4c (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit 93b29716cdce (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit ff6a5120e7ea (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit 6fd6ae533248 (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit def7ba03a1af (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit 08871f240b3a (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit 2f039db377b2 (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit 7345bac74b29 (Drop more @errp parameters after previous 
commit)
16/18 Checking commit fdb540093c5d (qdev: Unrealize must not fail)
17/18 Checking commit f308ad47bac3 (spapr_pci: Drop some dead error handling)
18/18 Checking commit b6bd21f2445d (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[PATCH v4 12/14] block/crypto: implement blockdev-amend

[Maxim Levitsky]

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/crypto.c   | 72 
 qapi/block-core.json | 14 -
 2 files changed, 66 insertions(+), 20 deletions(-)

diff --git a/block/crypto.c b/block/crypto.c
index b71e57f777..d7725df79e 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -775,32 +775,21 @@ block_crypto_get_specific_info_luks(BlockDriverState *bs, 
Error **errp)
 }
 
 static int
-block_crypto_amend_options_luks(BlockDriverState *bs,
-QemuOpts *opts,
-BlockDriverAmendStatusCB *status_cb,
-void *cb_opaque,
-bool force,
-Error **errp)
+block_crypto_amend_options_generic_luks(BlockDriverState *bs,
+QCryptoBlockAmendOptions 
*amend_options,
+bool force,
+Error **errp)
 {
 BlockCrypto *crypto = bs->opaque;
-QDict *cryptoopts = NULL;
-QCryptoBlockAmendOptions *amend_options = NULL;
 int ret;
 
 assert(crypto);
 assert(crypto->block);
-crypto->updating_keys = true;
 
+/* apply for exclusive read/write permissions to the underlying file*/
+crypto->updating_keys = true;
 ret = bdrv_child_refresh_perms(bs, bs->file, errp);
-if (ret < 0) {
-goto cleanup;
-}
-
-cryptoopts = qemu_opts_to_qdict(opts, NULL);
-qdict_put_str(cryptoopts, "format", "luks");
-amend_options = block_crypto_amend_opts_init(cryptoopts, errp);
-if (!amend_options) {
-ret = -EINVAL;
+if (ret) {
 goto cleanup;
 }
 
@@ -812,13 +801,57 @@ block_crypto_amend_options_luks(BlockDriverState *bs,
   force,
   errp);
 cleanup:
+/* release exclusive read/write permissions to the underlying file*/
 crypto->updating_keys = false;
 bdrv_child_refresh_perms(bs, bs->file, errp);
-qapi_free_QCryptoBlockAmendOptions(amend_options);
+return ret;
+}
+
+static int
+block_crypto_amend_options_luks(BlockDriverState *bs,
+QemuOpts *opts,
+BlockDriverAmendStatusCB *status_cb,
+void *cb_opaque,
+bool force,
+Error **errp)
+{
+BlockCrypto *crypto = bs->opaque;
+QDict *cryptoopts = NULL;
+QCryptoBlockAmendOptions *amend_options = NULL;
+int ret = -EINVAL;
+
+assert(crypto);
+assert(crypto->block);
+
+cryptoopts = qemu_opts_to_qdict(opts, NULL);
+qdict_put_str(cryptoopts, "format", "luks");
+amend_options = block_crypto_amend_opts_init(cryptoopts, errp);
 qobject_unref(cryptoopts);
+if (!amend_options) {
+goto cleanup;
+}
+ret = block_crypto_amend_options_generic_luks(bs, amend_options,
+  force, errp);
+cleanup:
+qapi_free_QCryptoBlockAmendOptions(amend_options);
 return ret;
 }
 
+static int
+coroutine_fn block_crypto_co_amend_luks(BlockDriverState *bs,
+BlockdevAmendOptions *opts,
+bool force,
+Error **errp)
+{
+QCryptoBlockAmendOptions amend_opts;
+
+amend_opts = (QCryptoBlockAmendOptions) {
+.format = Q_CRYPTO_BLOCK_FORMAT_LUKS,
+.u.luks = *qapi_BlockdevAmendOptionsLUKS_base(>u.luks),
+};
+return block_crypto_amend_options_generic_luks(bs, _opts,
+   force, errp);
+}
 
 static void
 block_crypto_child_perms(BlockDriverState *bs, BdrvChild *c,
@@ -891,6 +924,7 @@ static BlockDriver bdrv_crypto_luks = {
 .bdrv_get_info  = block_crypto_get_info_luks,
 .bdrv_get_specific_info = block_crypto_get_specific_info_luks,
 .bdrv_amend_options = block_crypto_amend_options_luks,
+.bdrv_co_amend  = block_crypto_co_amend_luks,
 
 .strong_runtime_opts = block_crypto_strong_runtime_opts,
 };
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 5b9123c15f..a5f679ac17 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4649,6 +4649,18 @@
   'data': { 'job-id': 'str',
 'options': 'BlockdevCreateOptions' } }
 
+##
+# @BlockdevAmendOptionsLUKS:
+#
+# Driver specific image amend options for LUKS.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockdevAmendOptionsLUKS',
+  'base': 'QCryptoBlockAmendOptionsLUKS',
+  'data': { }
+}
+
 ##
 # @BlockdevAmendOptions:
 #
@@ -4663,7 +4675,7 @@
   'driver': 'BlockdevDriver' },
   'discriminator': 'driver',
   'data': {
-  } }
+  'luks':   'BlockdevAmendOptionsLUKS' } }
 
 ##
 # @x-blockdev-amend:
-- 
2.17.2

[PATCH v4 11/14] block/core: add generic infrastructure for x-blockdev-amend qmp command

[Maxim Levitsky]

blockdev-amend will be used similiar to blockdev-create
to allow on the fly changes of the structure of the format based block devices.

Current plan is to first support encryption keyslot management for luks
based formats (raw and embedded in qcow2)

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/Makefile.objs   |   2 +-
 block/amend.c | 108 ++
 include/block/block_int.h |  21 +---
 qapi/block-core.json  |  42 +++
 qapi/job.json |   4 +-
 5 files changed, 169 insertions(+), 8 deletions(-)
 create mode 100644 block/amend.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 3635b6b4c1..a0988638d5 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -19,7 +19,7 @@ block-obj-$(CONFIG_WIN32) += file-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += file-posix.o
 block-obj-$(CONFIG_LINUX_AIO) += linux-aio.o
 block-obj-$(CONFIG_LINUX_IO_URING) += io_uring.o
-block-obj-y += null.o mirror.o commit.o io.o create.o
+block-obj-y += null.o mirror.o commit.o io.o create.o amend.o
 block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
diff --git a/block/amend.c b/block/amend.c
new file mode 100644
index 00..2db7b1eafc
--- /dev/null
+++ b/block/amend.c
@@ -0,0 +1,108 @@
+/*
+ * Block layer code related to image options amend
+ *
+ * Copyright (c) 2018 Kevin Wolf 
+ * Copyright (c) 2019 Maxim Levitsky 
+ *
+ * Heavily based on create.c
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "qemu/job.h"
+#include "qemu/main-loop.h"
+#include "qapi/qapi-commands-block-core.h"
+#include "qapi/qapi-visit-block-core.h"
+#include "qapi/clone-visitor.h"
+#include "qapi/error.h"
+
+typedef struct BlockdevAmendJob {
+Job common;
+BlockdevAmendOptions *opts;
+BlockDriverState *bs;
+bool force;
+} BlockdevAmendJob;
+
+static int coroutine_fn blockdev_amend_run(Job *job, Error **errp)
+{
+BlockdevAmendJob *s = container_of(job, BlockdevAmendJob, common);
+int ret;
+
+job_progress_set_remaining(>common, 1);
+ret = s->bs->drv->bdrv_co_amend(s->bs, s->opts, s->force, errp);
+job_progress_update(>common, 1);
+qapi_free_BlockdevAmendOptions(s->opts);
+return ret;
+}
+
+static const JobDriver blockdev_amend_job_driver = {
+.instance_size = sizeof(BlockdevAmendJob),
+.job_type  = JOB_TYPE_AMEND,
+.run   = blockdev_amend_run,
+};
+
+void qmp_x_blockdev_amend(const char *job_id,
+  const char *node_name,
+  BlockdevAmendOptions *options,
+  bool has_force,
+  bool force,
+  Error **errp)
+{
+BlockdevAmendJob *s;
+const char *fmt = BlockdevDriver_str(options->driver);
+BlockDriver *drv = bdrv_find_format(fmt);
+BlockDriverState *bs = bdrv_find_node(node_name);
+
+/*
+ * If the driver is in the schema, we know that it exists. But it may not
+ * be whitelisted.
+ */
+assert(drv);
+if (bdrv_uses_whitelist() && !bdrv_is_whitelisted(drv, false)) {
+error_setg(errp, "Driver is not whitelisted");
+return;
+}
+
+if (bs->drv != drv) {
+error_setg(errp,
+   "x-blockdev-amend doesn't support changing the block 
driver");
+return;
+}
+
+/* Error out if the driver doesn't support .bdrv_co_amend */
+if (!drv->bdrv_co_amend) {
+error_setg(errp, "Driver does not support x-blockdev-amend");
+return;
+}
+
+/* Create the block job */
+s = job_create(job_id, _amend_job_driver, NULL,
+   bdrv_get_aio_context(bs), JOB_DEFAULT | JOB_MANUAL_DISMISS,
+   NULL, NULL, errp);
+if (!s) {
+return;
+}
+
+s->bs = bs,
+s->opts = 

[PATCH v4 13/14] block/qcow2: implement blockdev-amend

[Maxim Levitsky]

Currently the implementation only supports amending the encryption
options, unlike the qemu-img version

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/qcow2.c| 39 +++
 qapi/block-core.json | 16 +++-
 2 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index ce1e25f341..a770b88a8f 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -5448,6 +5448,44 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 return 0;
 }
 
+static int coroutine_fn qcow2_co_amend(BlockDriverState *bs,
+   BlockdevAmendOptions *opts,
+   bool force,
+   Error **errp)
+{
+BlockdevAmendOptionsQcow2 *qopts = >u.qcow2;
+BDRVQcow2State *s = bs->opaque;
+int ret = 0;
+
+if (qopts->has_encrypt) {
+if (!s->crypto) {
+error_setg(errp, "image is not encrypted, can't amend");
+return -EOPNOTSUPP;
+}
+
+if (qopts->encrypt->format != Q_CRYPTO_BLOCK_FORMAT_LUKS) {
+error_setg(errp,
+   "Amend can't be used to change the qcow2 encryption 
format");
+return -EOPNOTSUPP;
+}
+
+if (s->crypt_method_header != QCOW_CRYPT_LUKS) {
+error_setg(errp,
+   "Only LUKS encryption options can be amended for qcow2 
with blockdev-amend");
+return -EOPNOTSUPP;
+}
+
+ret = qcrypto_block_amend_options(s->crypto,
+  qcow2_crypto_hdr_read_func,
+  qcow2_crypto_hdr_write_func,
+  bs,
+  qopts->encrypt,
+  force,
+  errp);
+}
+return ret;
+}
+
 /*
  * If offset or size are negative, respectively, they will not be included in
  * the BLOCK_IMAGE_CORRUPTED event emitted.
@@ -5658,6 +5696,7 @@ BlockDriver bdrv_qcow2 = {
 .mutable_opts= mutable_opts,
 .bdrv_co_check   = qcow2_co_check,
 .bdrv_amend_options  = qcow2_amend_options,
+.bdrv_co_amend   = qcow2_co_amend,
 
 .bdrv_detach_aio_context  = qcow2_detach_aio_context,
 .bdrv_attach_aio_context  = qcow2_attach_aio_context,
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a5f679ac17..0ffdc1c3d4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4661,6 +4661,19 @@
   'data': { }
 }
 
+##
+# @BlockdevAmendOptionsQcow2:
+#
+# Driver specific image amend options for qcow2.
+# For now, only encryption options can be amended
+#
+# @encrypt  Encryption options to be amended
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockdevAmendOptionsQcow2',
+  'data': { '*encrypt': 'QCryptoBlockAmendOptions' } }
+
 ##
 # @BlockdevAmendOptions:
 #
@@ -4675,7 +4688,8 @@
   'driver': 'BlockdevDriver' },
   'discriminator': 'driver',
   'data': {
-  'luks':   'BlockdevAmendOptionsLUKS' } }
+  'luks':   'BlockdevAmendOptionsLUKS',
+  'qcow2':  'BlockdevAmendOptionsQcow2' } }
 
 ##
 # @x-blockdev-amend:
-- 
2.17.2

[PATCH v4 08/14] block/qcow2: extend qemu-img amend interface with crypto options

[Maxim Levitsky]

Now that we have all the infrastructure in place,
wire it in the qcow2 driver and expose this to the user.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/qcow2.c  | 72 +-
 tests/qemu-iotests/082.out | 45 
 2 files changed, 108 insertions(+), 9 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index e6c4d0b0b4..ce1e25f341 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -176,6 +176,19 @@ static ssize_t qcow2_crypto_hdr_write_func(QCryptoBlock 
*block, size_t offset,
 return ret;
 }
 
+static QDict*
+qcow2_extract_crypto_opts(QemuOpts *opts, const char *fmt, Error **errp)
+{
+QDict *cryptoopts_qdict;
+QDict *opts_qdict;
+
+/* Extract "encrypt." options into a qdict */
+opts_qdict = qemu_opts_to_qdict(opts, NULL);
+qdict_extract_subqdict(opts_qdict, _qdict, "encrypt.");
+qobject_unref(opts_qdict);
+qdict_put_str(cryptoopts_qdict, "format", fmt);
+return cryptoopts_qdict;
+}
 
 /*
  * read qcow2 extension and fill bs
@@ -4733,17 +4746,11 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, 
BlockDriverState *in_bs,
 g_free(optstr);
 
 if (has_luks) {
+
 g_autoptr(QCryptoBlockCreateOptions) create_opts = NULL;
-QDict *opts_qdict;
-QDict *cryptoopts;
+QDict *cryptoopts = qcow2_extract_crypto_opts(opts, "luks", errp);
 size_t headerlen;
 
-opts_qdict = qemu_opts_to_qdict(opts, NULL);
-qdict_extract_subqdict(opts_qdict, , "encrypt.");
-qobject_unref(opts_qdict);
-
-qdict_put_str(cryptoopts, "format", "luks");
-
 create_opts = block_crypto_create_opts_init(cryptoopts, errp);
 qobject_unref(cryptoopts);
 if (!create_opts) {
@@ -5122,6 +5129,7 @@ typedef enum Qcow2AmendOperation {
 QCOW2_NO_OPERATION = 0,
 
 QCOW2_UPGRADING,
+QCOW2_UPDATING_ENCRYPTION,
 QCOW2_CHANGING_REFCOUNT_ORDER,
 QCOW2_DOWNGRADING,
 } Qcow2AmendOperation;
@@ -5203,6 +5211,7 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 int ret;
 QemuOptDesc *desc = opts->list->desc;
 Qcow2AmendHelperCBInfo helper_cb_info;
+bool encryption_update = false;
 
 while (desc && desc->name) {
 if (!qemu_opt_find(opts, desc->name)) {
@@ -5229,6 +5238,18 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 backing_file = qemu_opt_get(opts, BLOCK_OPT_BACKING_FILE);
 } else if (!strcmp(desc->name, BLOCK_OPT_BACKING_FMT)) {
 backing_format = qemu_opt_get(opts, BLOCK_OPT_BACKING_FMT);
+} else if (g_str_has_prefix(desc->name, "encrypt.")) {
+if (!s->crypto) {
+error_setg(errp,
+   "Can't amend encryption options - encryption not 
present");
+return -EINVAL;
+}
+if (s->crypt_method_header != QCOW_CRYPT_LUKS) {
+error_setg(errp,
+   "Only LUKS encryption options can be amended");
+return -ENOTSUP;
+}
+encryption_update = true;
 } else if (!strcmp(desc->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
 lazy_refcounts = qemu_opt_get_bool(opts, BLOCK_OPT_LAZY_REFCOUNTS,
lazy_refcounts);
@@ -5271,7 +5292,8 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 .original_status_cb = status_cb,
 .original_cb_opaque = cb_opaque,
 .total_operations = (new_version != old_version)
-  + (s->refcount_bits != refcount_bits)
+  + (s->refcount_bits != refcount_bits) +
+(encryption_update == true)
 };
 
 /* Upgrade first (some features may require compat=1.1) */
@@ -5284,6 +5306,33 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 }
 }
 
+if (encryption_update) {
+QDict *amend_opts_dict;
+QCryptoBlockAmendOptions *amend_opts;
+
+helper_cb_info.current_operation = QCOW2_UPDATING_ENCRYPTION;
+amend_opts_dict = qcow2_extract_crypto_opts(opts, "luks", errp);
+if (!amend_opts_dict) {
+return -EINVAL;
+}
+amend_opts = block_crypto_amend_opts_init(amend_opts_dict, errp);
+qobject_unref(amend_opts_dict);
+if (!amend_opts) {
+return -EINVAL;
+}
+ret = qcrypto_block_amend_options(s->crypto,
+  qcow2_crypto_hdr_read_func,
+  qcow2_crypto_hdr_write_func,
+  bs,
+  amend_opts,
+  force,
+  errp);
+qapi_free_QCryptoBlockAmendOptions(amend_opts);
+if (ret < 0) {
+

[PATCH v4 09/14] iotests: filter few more luks specific create options

[Maxim Levitsky]

This allows more tests to be able to have same output on both qcow2 luks 
encrypted images
and raw luks images

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 tests/qemu-iotests/087.out   |  6 ++---
 tests/qemu-iotests/134.out   |  2 +-
 tests/qemu-iotests/158.out   |  4 +--
 tests/qemu-iotests/188.out   |  2 +-
 tests/qemu-iotests/189.out   |  4 +--
 tests/qemu-iotests/198.out   |  4 +--
 tests/qemu-iotests/263.out   |  4 +--
 tests/qemu-iotests/274.out   | 46 
 tests/qemu-iotests/284.out   |  6 ++---
 tests/qemu-iotests/common.filter |  6 +++--
 10 files changed, 43 insertions(+), 41 deletions(-)

diff --git a/tests/qemu-iotests/087.out b/tests/qemu-iotests/087.out
index f23bffbbf1..d5ff53302e 100644
--- a/tests/qemu-iotests/087.out
+++ b/tests/qemu-iotests/087.out
@@ -34,7 +34,7 @@ QMP_VERSION
 
 === Encrypted image QCow ===
 
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on 
encrypt.key-secret=sec0 size=134217728
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on size=134217728
 Testing:
 QMP_VERSION
 {"return": {}}
@@ -46,7 +46,7 @@ QMP_VERSION
 
 === Encrypted image LUKS ===
 
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encrypt.format=luks 
encrypt.key-secret=sec0 size=134217728
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 Testing:
 QMP_VERSION
 {"return": {}}
@@ -58,7 +58,7 @@ QMP_VERSION
 
 === Missing driver ===
 
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on 
encrypt.key-secret=sec0 size=134217728
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on size=134217728
 Testing: -S
 QMP_VERSION
 {"return": {}}
diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
index f2878f5f3a..e4733c0b81 100644
--- a/tests/qemu-iotests/134.out
+++ b/tests/qemu-iotests/134.out
@@ -1,5 +1,5 @@
 QA output created by 134
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on 
encrypt.key-secret=sec0 size=134217728
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on size=134217728
 
 == reading whole image ==
 read 134217728/134217728 bytes at offset 0
diff --git a/tests/qemu-iotests/158.out b/tests/qemu-iotests/158.out
index fa2294bb85..52ea9a488f 100644
--- a/tests/qemu-iotests/158.out
+++ b/tests/qemu-iotests/158.out
@@ -1,6 +1,6 @@
 QA output created by 158
 == create base ==
-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT encryption=on 
encrypt.key-secret=sec0 size=134217728
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT encryption=on size=134217728
 
 == writing whole image ==
 wrote 134217728/134217728 bytes at offset 0
@@ -10,7 +10,7 @@ wrote 134217728/134217728 bytes at offset 0
 read 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 == create overlay ==
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on 
encrypt.key-secret=sec0 size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encryption=on size=134217728 
backing_file=TEST_DIR/t.IMGFMT.base
 
 == writing part of a cluster ==
 wrote 1024/1024 bytes at offset 0
diff --git a/tests/qemu-iotests/188.out b/tests/qemu-iotests/188.out
index 4b9aadd51c..5426861b18 100644
--- a/tests/qemu-iotests/188.out
+++ b/tests/qemu-iotests/188.out
@@ -1,5 +1,5 @@
 QA output created by 188
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10 size=16777216
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216
 
 == reading whole image ==
 read 16777216/16777216 bytes at offset 0
diff --git a/tests/qemu-iotests/189.out b/tests/qemu-iotests/189.out
index e536d95d53..bc213cbe14 100644
--- a/tests/qemu-iotests/189.out
+++ b/tests/qemu-iotests/189.out
@@ -1,6 +1,6 @@
 QA output created by 189
 == create base ==
-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10 size=16777216
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=16777216
 
 == writing whole image ==
 wrote 16777216/16777216 bytes at offset 0
@@ -10,7 +10,7 @@ wrote 16777216/16777216 bytes at offset 0
 read 16777216/16777216 bytes at offset 0
 16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 == create overlay ==
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT encrypt.format=luks 
encrypt.key-secret=sec1 encrypt.iter-time=10 size=16777216 
backing_file=TEST_DIR/t.IMGFMT.base
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 
backing_file=TEST_DIR/t.IMGFMT.base
 
 == writing part of a cluster ==
 wrote 1024/1024 bytes at offset 0
diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out
index b0f2d417af..acfdf96b0c 100644
--- a/tests/qemu-iotests/198.out
+++ b/tests/qemu-iotests/198.out
@@ -1,12 +1,12 @@
 QA output created by 198
 == create base ==
-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10 size=16777216
+Formatting 'TEST_DIR/t.IMGFMT.base', 

[PATCH v4 14/14] iotests: add tests for blockdev-amend

[Maxim Levitsky]

This commit adds two tests that cover the
new blockdev-amend functionality of luks and qcow2 driver

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 tests/qemu-iotests/302 | 278 +
 tests/qemu-iotests/302.out |  40 ++
 tests/qemu-iotests/303 | 233 +++
 tests/qemu-iotests/303.out |  33 +
 tests/qemu-iotests/group   |   2 +
 5 files changed, 586 insertions(+)
 create mode 100755 tests/qemu-iotests/302
 create mode 100644 tests/qemu-iotests/302.out
 create mode 100755 tests/qemu-iotests/303
 create mode 100644 tests/qemu-iotests/303.out

diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302
new file mode 100755
index 00..f7b4d13bd2
--- /dev/null
+++ b/tests/qemu-iotests/302
@@ -0,0 +1,278 @@
+#!/usr/bin/env python3
+#
+# Test case QMP's encrypted key management
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import iotests
+import os
+import time
+import json
+
+test_img = os.path.join(iotests.test_dir, 'test.img')
+
+class Secret:
+def __init__(self, index):
+self._id = "keysec" + str(index)
+# you are not supposed to see the password...
+self._secret = "hunter" + str(index)
+
+def id(self):
+return self._id
+
+def secret(self):
+return self._secret
+
+def to_cmdline_object(self):
+return  [ "secret,id=" + self._id + ",data=" + self._secret]
+
+def to_qmp_object(self):
+return { "qom_type" : "secret", "id": self.id(),
+ "props": { "data": self.secret() } }
+
+
+class EncryptionSetupTestCase(iotests.QMPTestCase):
+
+# test case startup
+def setUp(self):
+# start the VM
+self.vm = iotests.VM()
+self.vm.launch()
+
+# create the secrets and load 'em into the VM
+self.secrets = [ Secret(i) for i in range(0, 6) ]
+for secret in self.secrets:
+result = self.vm.qmp("object-add", **secret.to_qmp_object())
+self.assert_qmp(result, 'return', {})
+
+if iotests.imgfmt == "qcow2":
+self.pfx = "encrypt."
+self.img_opts = [ '-o', "encrypt.format=luks" ]
+else:
+self.pfx = ""
+self.img_opts = []
+
+# test case shutdown
+def tearDown(self):
+# stop the VM
+self.vm.shutdown()
+
+###
+# create the encrypted block device
+def createImg(self, file, secret):
+
+iotests.qemu_img(
+'create',
+'--object', *secret.to_cmdline_object(),
+'-f', iotests.imgfmt,
+'-o', self.pfx + 'key-secret=' + secret.id(),
+'-o', self.pfx + 'iter-time=10',
+*self.img_opts,
+file,
+'1M')
+
+###
+# open an encrypted block device
+def openImageQmp(self, id, file, secret, read_only = False):
+
+encrypt_options = {
+'key-secret' : secret.id()
+}
+
+if iotests.imgfmt == "qcow2":
+encrypt_options = {
+'encrypt': {
+'format':'luks',
+**encrypt_options
+}
+}
+
+result = self.vm.qmp('blockdev-add', **
+{
+'driver': iotests.imgfmt,
+'node-name': id,
+'read-only': read_only,
+
+**encrypt_options,
+
+'file': {
+'driver': 'file',
+'filename': test_img,
+}
+}
+)
+self.assert_qmp(result, 'return', {})
+
+# close the encrypted block device
+def closeImageQmp(self, id):
+result = self.vm.qmp('blockdev-del', **{ 'node-name': id })
+self.assert_qmp(result, 'return', {})
+
+###
+# add a key to an encrypted block device
+def addKeyQmp(self, id, new_secret, secret = None,
+  slot = None, force = False):
+
+crypt_options = {
+'state'  : 'active',
+'new-secret' : 

[PATCH v4 10/14] iotests: qemu-img tests for luks key management

[Maxim Levitsky]

This commit adds two tests, which test the new amend interface
of both luks raw images and qcow2 luks encrypted images.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 tests/qemu-iotests/300 | 207 +
 tests/qemu-iotests/300.out |  99 ++
 tests/qemu-iotests/301 |  90 
 tests/qemu-iotests/301.out |  30 ++
 tests/qemu-iotests/group   |   3 +
 5 files changed, 429 insertions(+)
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
 create mode 100755 tests/qemu-iotests/301
 create mode 100644 tests/qemu-iotests/301.out

diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
new file mode 100755
index 00..aa1a77690f
--- /dev/null
+++ b/tests/qemu-iotests/300
@@ -0,0 +1,207 @@
+#!/usr/bin/env bash
+#
+# Test encryption key management with luks
+# Based on 134
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=mlevi...@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2 luks
+_supported_proto file #TODO
+
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+if [ "$IMGFMT" = "qcow2" ] ; then
+   PR="encrypt."
+   EXTRA_IMG_ARGS="-o encrypt.format=luks"
+fi
+
+
+# secrets: you are supposed to see the password as ***, see :-)
+S0="--object secret,id=sec0,data=hunter0"
+S1="--object secret,id=sec1,data=hunter1"
+S2="--object secret,id=sec2,data=hunter2"
+S3="--object secret,id=sec3,data=hunter3"
+S4="--object secret,id=sec4,data=hunter4"
+SECRETS="$S0 $S1 $S2 $S3 $S4"
+
+# image with given secret
+IMGS0="--image-opts 
driver=$IMGFMT,file.filename=$TEST_IMG,${PR}key-secret=sec0"
+IMGS1="--image-opts 
driver=$IMGFMT,file.filename=$TEST_IMG,${PR}key-secret=sec1"
+IMGS2="--image-opts 
driver=$IMGFMT,file.filename=$TEST_IMG,${PR}key-secret=sec2"
+IMGS3="--image-opts 
driver=$IMGFMT,file.filename=$TEST_IMG,${PR}key-secret=sec3"
+IMGS4="--image-opts 
driver=$IMGFMT,file.filename=$TEST_IMG,${PR}key-secret=sec4"
+
+
+echo "== creating a test image =="
+_make_test_img $S0 $EXTRA_IMG_ARGS -o ${PR}key-secret=sec0,${PR}iter-time=10 
32M
+
+echo
+echo "== test that key 0 opens the image =="
+$QEMU_IO $S0 -c "read 0 4096" $IMGS0 | _filter_qemu_io | _filter_testdir
+
+echo
+echo "== adding a password to slot 4 =="
+$QEMU_IMG amend $SECRETS $IMGS0 -o 
${PR}state=active,${PR}new-secret=sec4,${PR}iter-time=10,${PR}keyslot=4
+echo "== adding a password to slot 1 =="
+$QEMU_IMG amend $SECRETS $IMGS0 -o 
${PR}state=active,${PR}new-secret=sec1,${PR}iter-time=10
+echo "== adding a password to slot 3 =="
+$QEMU_IMG amend $SECRETS $IMGS1 -o 
${PR}state=active,${PR}new-secret=sec3,${PR}iter-time=10,${PR}keyslot=3
+
+echo "== adding a password to slot 2 =="
+$QEMU_IMG amend $SECRETS $IMGS3 -o 
${PR}state=active,${PR}new-secret=sec2,${PR}iter-time=10
+
+
+echo "== erase slot 4 =="
+$QEMU_IMG amend $SECRETS $IMGS1 -o ${PR}state=inactive,${PR}keyslot=4 | 
_filter_img_create
+
+
+echo
+echo "== all secrets should work =="
+for IMG in "$IMGS0" "$IMGS1" "$IMGS2" "$IMGS3"; do
+   $QEMU_IO $SECRETS -c "read 0 4096" $IMG | _filter_qemu_io | 
_filter_testdir
+done
+
+echo
+echo "== erase slot 0 and try it =="
+$QEMU_IMG amend $SECRETS $IMGS1 -o ${PR}state=inactive,${PR}old-secret=sec0 | 
_filter_img_create
+$QEMU_IO $SECRETS -c "read 0 4096" $IMGS0 | _filter_qemu_io | _filter_testdir
+
+echo
+echo "== erase slot 2 and try it =="
+$QEMU_IMG amend $SECRETS $IMGS1 -o ${PR}state=inactive,${PR}keyslot=2 | 
_filter_img_create
+$QEMU_IO $SECRETS -c "read 0 4096" $IMGS2 | _filter_qemu_io | _filter_testdir
+
+
+# at this point slots 1 and 3 should be active
+
+echo
+echo "== filling  4 slots with secret 2 =="
+for i in $(seq 0 3) ; do
+   $QEMU_IMG amend $SECRETS $IMGS3 -o 
${PR}state=active,${PR}new-secret=sec2,${PR}iter-time=10
+done
+
+echo
+echo "== adding secret 0 =="
+   $QEMU_IMG amend $SECRETS $IMGS3 -o 
${PR}state=active,${PR}new-secret=sec0,${PR}iter-time=10
+
+echo
+echo "== adding secret 3 (last slot) =="
+   $QEMU_IMG amend $SECRETS $IMGS3 -o 

[PATCH v4 06/14] block/crypto: rename two functions

[Maxim Levitsky]

rename the write_func to create_write_func, and init_func to create_init_func.
This is preparation for other write_func that will be used to update the 
encryption keys.

No functional changes

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/crypto.c | 25 -
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/block/crypto.c b/block/crypto.c
index d379e39efb..13ca1ad891 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -79,12 +79,12 @@ struct BlockCryptoCreateData {
 };
 
 
-static ssize_t block_crypto_write_func(QCryptoBlock *block,
-   size_t offset,
-   const uint8_t *buf,
-   size_t buflen,
-   void *opaque,
-   Error **errp)
+static ssize_t block_crypto_create_write_func(QCryptoBlock *block,
+  size_t offset,
+  const uint8_t *buf,
+  size_t buflen,
+  void *opaque,
+  Error **errp)
 {
 struct BlockCryptoCreateData *data = opaque;
 ssize_t ret;
@@ -97,11 +97,10 @@ static ssize_t block_crypto_write_func(QCryptoBlock *block,
 return ret;
 }
 
-
-static ssize_t block_crypto_init_func(QCryptoBlock *block,
-  size_t headerlen,
-  void *opaque,
-  Error **errp)
+static ssize_t block_crypto_create_init_func(QCryptoBlock *block,
+ size_t headerlen,
+ void *opaque,
+ Error **errp)
 {
 struct BlockCryptoCreateData *data = opaque;
 
@@ -297,8 +296,8 @@ static int block_crypto_co_create_generic(BlockDriverState 
*bs,
 };
 
 crypto = qcrypto_block_create(opts, NULL,
-  block_crypto_init_func,
-  block_crypto_write_func,
+  block_crypto_create_init_func,
+  block_crypto_create_write_func,
   ,
   errp);
 
-- 
2.17.2

[PATCH v4 07/14] block/crypto: implement the encryption key management

[Maxim Levitsky]

This implements the encryption key management using the generic code in
qcrypto layer and exposes it to the user via qemu-img

This code adds another 'write_func' because the initialization
write_func works directly on the underlying file, and amend
works on instance of luks device.

This commit also adds a 'hack/workaround' I and Kevin Wolf (thanks)
made to make the driver both support write sharing (to avoid breaking the 
users),
and be safe against concurrent  metadata update (the keyslots)

Eventually the write sharing for luks driver will be deprecated
and removed together with this hack.

The hack is that we ask (as a format driver) for BLK_PERM_CONSISTENT_READ
and then when we want to update the keys, we unshare that permission.
So if someone else has the image open, even readonly, encryption
key update will fail gracefully.

Also thanks to Daniel Berrange for the idea of
unsharing read, rather that write permission which allows
to avoid cases when the other user had opened the image read-only.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/crypto.c | 127 +++--
 block/crypto.h |  34 +
 2 files changed, 158 insertions(+), 3 deletions(-)

diff --git a/block/crypto.c b/block/crypto.c
index 13ca1ad891..b71e57f777 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -37,6 +37,7 @@ typedef struct BlockCrypto BlockCrypto;
 
 struct BlockCrypto {
 QCryptoBlock *block;
+bool updating_keys;
 };
 
 
@@ -71,6 +72,24 @@ static ssize_t block_crypto_read_func(QCryptoBlock *block,
 return ret;
 }
 
+static ssize_t block_crypto_write_func(QCryptoBlock *block,
+   size_t offset,
+   const uint8_t *buf,
+   size_t buflen,
+   void *opaque,
+   Error **errp)
+{
+BlockDriverState *bs = opaque;
+ssize_t ret;
+
+ret = bdrv_pwrite(bs->file, offset, buf, buflen);
+if (ret < 0) {
+error_setg_errno(errp, -ret, "Could not write encryption header");
+return ret;
+}
+return ret;
+}
+
 
 struct BlockCryptoCreateData {
 BlockBackend *blk;
@@ -149,6 +168,19 @@ static QemuOptsList block_crypto_create_opts_luks = {
 };
 
 
+static QemuOptsList block_crypto_amend_opts_luks = {
+.name = "crypto",
+.head = QTAILQ_HEAD_INITIALIZER(block_crypto_create_opts_luks.head),
+.desc = {
+BLOCK_CRYPTO_OPT_DEF_LUKS_STATE(""),
+BLOCK_CRYPTO_OPT_DEF_LUKS_KEYSLOT(""),
+BLOCK_CRYPTO_OPT_DEF_LUKS_OLD_SECRET(""),
+BLOCK_CRYPTO_OPT_DEF_LUKS_NEW_SECRET(""),
+BLOCK_CRYPTO_OPT_DEF_LUKS_ITER_TIME(""),
+{ /* end of list */ }
+},
+};
+
 QCryptoBlockOpenOptions *
 block_crypto_open_opts_init(QDict *opts, Error **errp)
 {
@@ -742,6 +774,95 @@ block_crypto_get_specific_info_luks(BlockDriverState *bs, 
Error **errp)
 return spec_info;
 }
 
+static int
+block_crypto_amend_options_luks(BlockDriverState *bs,
+QemuOpts *opts,
+BlockDriverAmendStatusCB *status_cb,
+void *cb_opaque,
+bool force,
+Error **errp)
+{
+BlockCrypto *crypto = bs->opaque;
+QDict *cryptoopts = NULL;
+QCryptoBlockAmendOptions *amend_options = NULL;
+int ret;
+
+assert(crypto);
+assert(crypto->block);
+crypto->updating_keys = true;
+
+ret = bdrv_child_refresh_perms(bs, bs->file, errp);
+if (ret < 0) {
+goto cleanup;
+}
+
+cryptoopts = qemu_opts_to_qdict(opts, NULL);
+qdict_put_str(cryptoopts, "format", "luks");
+amend_options = block_crypto_amend_opts_init(cryptoopts, errp);
+if (!amend_options) {
+ret = -EINVAL;
+goto cleanup;
+}
+
+ret = qcrypto_block_amend_options(crypto->block,
+  block_crypto_read_func,
+  block_crypto_write_func,
+  bs,
+  amend_options,
+  force,
+  errp);
+cleanup:
+crypto->updating_keys = false;
+bdrv_child_refresh_perms(bs, bs->file, errp);
+qapi_free_QCryptoBlockAmendOptions(amend_options);
+qobject_unref(cryptoopts);
+return ret;
+}
+
+
+static void
+block_crypto_child_perms(BlockDriverState *bs, BdrvChild *c,
+ const BdrvChildRole *role,
+ BlockReopenQueue *reopen_queue,
+ uint64_t perm, uint64_t shared,
+ uint64_t *nperm, uint64_t *nshared)
+{
+
+BlockCrypto *crypto = bs->opaque;
+
+bdrv_filter_default_perms(bs, c, role, reopen_queue,
+perm, shared, nperm, nshared);
+/*
+ * Ask for 

[PATCH v4 04/14] block/amend: separate amend and create options for qemu-img

[Maxim Levitsky]

Some options are only useful for creation
(or hard to be amended, like cluster size for qcow2), while some other
options are only useful for amend, like upcoming keyslot management
options for luks

Since currently only qcow2 supports amend, move all its options
to a common macro and then include it in each action option list.

In future it might be useful to remove some options which are
not supported anyway from amend list, which currently
cause an error message if amended.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/qcow2.c | 160 +-
 include/block/block_int.h |   4 +
 qemu-img.c|  18 ++---
 3 files changed, 100 insertions(+), 82 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index ffb6b22e2d..13780b0278 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -5495,83 +5495,96 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool 
fatal, int64_t offset,
 s->signaled_corruption = true;
 }
 
+#define QCOW_COMMON_OPTIONS \
+{   \
+.name = BLOCK_OPT_SIZE, \
+.type = QEMU_OPT_SIZE,  \
+.help = "Virtual disk size" \
+},  \
+{   \
+.name = BLOCK_OPT_COMPAT_LEVEL, \
+.type = QEMU_OPT_STRING,\
+.help = "Compatibility level (v2 [0.10] or v3 [1.1])"   \
+},  \
+{   \
+.name = BLOCK_OPT_BACKING_FILE, \
+.type = QEMU_OPT_STRING,\
+.help = "File name of a base image" \
+},  \
+{   \
+.name = BLOCK_OPT_BACKING_FMT,  \
+.type = QEMU_OPT_STRING,\
+.help = "Image format of the base image"\
+},  \
+{   \
+.name = BLOCK_OPT_DATA_FILE,\
+.type = QEMU_OPT_STRING,\
+.help = "File name of an external data file"\
+},  \
+{   \
+.name = BLOCK_OPT_DATA_FILE_RAW,\
+.type = QEMU_OPT_BOOL,  \
+.help = "The external data file must stay valid "   \
+"as a raw image"\
+},  \
+{   \
+.name = BLOCK_OPT_ENCRYPT,  \
+.type = QEMU_OPT_BOOL,  \
+.help = "Encrypt the image with format 'aes'. (Deprecated " \
+"in favor of " BLOCK_OPT_ENCRYPT_FORMAT "=aes)",\
+},  \
+{   \
+.name = BLOCK_OPT_ENCRYPT_FORMAT,   \
+.type = QEMU_OPT_STRING,\
+.help = "Encrypt the image, format choices: 'aes', 'luks'", \
+},  \
+BLOCK_CRYPTO_OPT_DEF_KEY_SECRET("encrypt.", \
+"ID of secret providing qcow AES key or LUKS passphrase"),  \
+BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_ALG("encrypt."),   \
+BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_MODE("encrypt."),  \
+BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_ALG("encrypt."),\
+BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_HASH_ALG("encrypt."),   \
+BLOCK_CRYPTO_OPT_DEF_LUKS_HASH_ALG("encrypt."), \
+BLOCK_CRYPTO_OPT_DEF_LUKS_ITER_TIME("encrypt."),\
+{   \
+.name = BLOCK_OPT_CLUSTER_SIZE, \
+.type = QEMU_OPT_SIZE,  \
+.help = "qcow2 cluster size",   \
+.def_value_str = stringify(DEFAULT_CLUSTER_SIZE)\
+},  

[PATCH v4 05/14] block/amend: refactor qcow2 amend options

[Maxim Levitsky]

Some qcow2 create options can't be used for amend.
Remove them from the qcow2 create options and add generic logic to detect
such options in qemu-img

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/qcow2.c  | 108 ++---
 qemu-img.c |  18 +++-
 tests/qemu-iotests/049.out | 102 ++--
 tests/qemu-iotests/061.out |  12 ++-
 tests/qemu-iotests/079.out |  18 ++--
 tests/qemu-iotests/082.out | 149 
 tests/qemu-iotests/085.out |  38 
 tests/qemu-iotests/087.out |   6 +-
 tests/qemu-iotests/115.out |   2 +-
 tests/qemu-iotests/121.out |   4 +-
 tests/qemu-iotests/125.out | 192 ++---
 tests/qemu-iotests/134.out |   2 +-
 tests/qemu-iotests/144.out |   4 +-
 tests/qemu-iotests/158.out |   4 +-
 tests/qemu-iotests/182.out |   2 +-
 tests/qemu-iotests/185.out |   8 +-
 tests/qemu-iotests/188.out |   2 +-
 tests/qemu-iotests/189.out |   4 +-
 tests/qemu-iotests/198.out |   4 +-
 tests/qemu-iotests/243.out |  16 ++--
 tests/qemu-iotests/250.out |   2 +-
 tests/qemu-iotests/255.out |   8 +-
 tests/qemu-iotests/259.out |   2 +-
 tests/qemu-iotests/263.out |   4 +-
 tests/qemu-iotests/280.out |   2 +-
 25 files changed, 284 insertions(+), 429 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 13780b0278..e6c4d0b0b4 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2971,17 +2971,6 @@ static int qcow2_change_backing_file(BlockDriverState 
*bs,
 return qcow2_update_header(bs);
 }
 
-static int qcow2_crypt_method_from_format(const char *encryptfmt)
-{
-if (g_str_equal(encryptfmt, "luks")) {
-return QCOW_CRYPT_LUKS;
-} else if (g_str_equal(encryptfmt, "aes")) {
-return QCOW_CRYPT_AES;
-} else {
-return -EINVAL;
-}
-}
-
 static int qcow2_set_up_encryption(BlockDriverState *bs,
QCryptoBlockCreateOptions *cryptoopts,
Error **errp)
@@ -5210,9 +5199,6 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 bool lazy_refcounts = s->use_lazy_refcounts;
 bool data_file_raw = data_file_is_raw(bs);
 const char *compat = NULL;
-uint64_t cluster_size = s->cluster_size;
-bool encrypt;
-int encformat;
 int refcount_bits = s->refcount_bits;
 int ret;
 QemuOptDesc *desc = opts->list->desc;
@@ -5237,44 +5223,12 @@ static int qcow2_amend_options(BlockDriverState *bs, 
QemuOpts *opts,
 error_setg(errp, "Unknown compatibility level %s", compat);
 return -EINVAL;
 }
-} else if (!strcmp(desc->name, BLOCK_OPT_PREALLOC)) {
-error_setg(errp, "Cannot change preallocation mode");
-return -ENOTSUP;
 } else if (!strcmp(desc->name, BLOCK_OPT_SIZE)) {
 new_size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, 0);
 } else if (!strcmp(desc->name, BLOCK_OPT_BACKING_FILE)) {
 backing_file = qemu_opt_get(opts, BLOCK_OPT_BACKING_FILE);
 } else if (!strcmp(desc->name, BLOCK_OPT_BACKING_FMT)) {
 backing_format = qemu_opt_get(opts, BLOCK_OPT_BACKING_FMT);
-} else if (!strcmp(desc->name, BLOCK_OPT_ENCRYPT)) {
-encrypt = qemu_opt_get_bool(opts, BLOCK_OPT_ENCRYPT,
-!!s->crypto);
-
-if (encrypt != !!s->crypto) {
-error_setg(errp,
-   "Changing the encryption flag is not supported");
-return -ENOTSUP;
-}
-} else if (!strcmp(desc->name, BLOCK_OPT_ENCRYPT_FORMAT)) {
-encformat = qcow2_crypt_method_from_format(
-qemu_opt_get(opts, BLOCK_OPT_ENCRYPT_FORMAT));
-
-if (encformat != s->crypt_method_header) {
-error_setg(errp,
-   "Changing the encryption format is not supported");
-return -ENOTSUP;
-}
-} else if (g_str_has_prefix(desc->name, "encrypt.")) {
-error_setg(errp,
-   "Changing the encryption parameters is not supported");
-return -ENOTSUP;
-} else if (!strcmp(desc->name, BLOCK_OPT_CLUSTER_SIZE)) {
-cluster_size = qemu_opt_get_size(opts, BLOCK_OPT_CLUSTER_SIZE,
- cluster_size);
-if (cluster_size != s->cluster_size) {
-error_setg(errp, "Changing the cluster size is not supported");
-return -ENOTSUP;
-}
 } else if (!strcmp(desc->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
 lazy_refcounts = qemu_opt_get_bool(opts, BLOCK_OPT_LAZY_REFCOUNTS,
lazy_refcounts);
@@ -5527,37 +5481,6 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool 
fatal, int64_t offset,
 .help = "The external data file must stay valid "   \

[PATCH v4 01/14] qcrypto/core: add generic infrastructure for crypto options amendment

[Maxim Levitsky]

This will be used first to implement luks keyslot management.

block_crypto_amend_opts_init will be used to convert
qemu-img cmdline to QCryptoBlockAmendOptions

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block/crypto.c | 17 +
 block/crypto.h |  3 +++
 crypto/block.c | 29 +
 crypto/blockpriv.h |  8 
 include/crypto/block.h | 22 ++
 qapi/crypto.json   | 16 
 6 files changed, 95 insertions(+)

diff --git a/block/crypto.c b/block/crypto.c
index e02f343590..d379e39efb 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -185,6 +185,23 @@ block_crypto_create_opts_init(QDict *opts, Error **errp)
 return ret;
 }
 
+QCryptoBlockAmendOptions *
+block_crypto_amend_opts_init(QDict *opts, Error **errp)
+{
+Visitor *v;
+QCryptoBlockAmendOptions *ret;
+
+v = qobject_input_visitor_new_flat_confused(opts, errp);
+if (!v) {
+return NULL;
+}
+
+visit_type_QCryptoBlockAmendOptions(v, NULL, , errp);
+
+visit_free(v);
+return ret;
+}
+
 
 static int block_crypto_open_generic(QCryptoBlockFormat format,
  QemuOptsList *opts_spec,
diff --git a/block/crypto.h b/block/crypto.h
index b935695e79..06e044c9be 100644
--- a/block/crypto.h
+++ b/block/crypto.h
@@ -91,6 +91,9 @@
 QCryptoBlockCreateOptions *
 block_crypto_create_opts_init(QDict *opts, Error **errp);
 
+QCryptoBlockAmendOptions *
+block_crypto_amend_opts_init(QDict *opts, Error **errp);
+
 QCryptoBlockOpenOptions *
 block_crypto_open_opts_init(QDict *opts, Error **errp);
 
diff --git a/crypto/block.c b/crypto/block.c
index 6f42b32f1e..eb057948b5 100644
--- a/crypto/block.c
+++ b/crypto/block.c
@@ -150,6 +150,35 @@ 
qcrypto_block_calculate_payload_offset(QCryptoBlockCreateOptions *create_opts,
 return crypto != NULL;
 }
 
+int qcrypto_block_amend_options(QCryptoBlock *block,
+QCryptoBlockReadFunc readfunc,
+QCryptoBlockWriteFunc writefunc,
+void *opaque,
+QCryptoBlockAmendOptions *options,
+bool force,
+Error **errp)
+{
+if (options->format != block->format) {
+error_setg(errp,
+   "Cannot amend encryption format");
+return -1;
+}
+
+if (!block->driver->amend) {
+error_setg(errp,
+   "Crypto format %s doesn't support format options amendment",
+   QCryptoBlockFormat_str(block->format));
+return -1;
+}
+
+return block->driver->amend(block,
+readfunc,
+writefunc,
+opaque,
+options,
+force,
+errp);
+}
 
 QCryptoBlockInfo *qcrypto_block_get_info(QCryptoBlock *block,
  Error **errp)
diff --git a/crypto/blockpriv.h b/crypto/blockpriv.h
index 71c59cb542..3c7ccea504 100644
--- a/crypto/blockpriv.h
+++ b/crypto/blockpriv.h
@@ -62,6 +62,14 @@ struct QCryptoBlockDriver {
   void *opaque,
   Error **errp);
 
+int (*amend)(QCryptoBlock *block,
+ QCryptoBlockReadFunc readfunc,
+ QCryptoBlockWriteFunc writefunc,
+ void *opaque,
+ QCryptoBlockAmendOptions *options,
+ bool force,
+ Error **errp);
+
 int (*get_info)(QCryptoBlock *block,
 QCryptoBlockInfo *info,
 Error **errp);
diff --git a/include/crypto/block.h b/include/crypto/block.h
index c77ccaf9c0..d274819791 100644
--- a/include/crypto/block.h
+++ b/include/crypto/block.h
@@ -144,6 +144,28 @@ QCryptoBlock 
*qcrypto_block_create(QCryptoBlockCreateOptions *options,
void *opaque,
Error **errp);
 
+/**
+ * qcrypto_block_amend_options:
+ * @block: the block encryption object
+ *
+ * @readfunc: callback for reading data from the volume header
+ * @writefunc: callback for writing data to the volume header
+ * @opaque: data to pass to @readfunc and @writefunc
+ * @options: the new/amended encryption options
+ * @force: hint for the driver to allow unsafe operation
+ * @errp: error pointer
+ *
+ * Changes the crypto options of the encryption format
+ *
+ */
+int qcrypto_block_amend_options(QCryptoBlock *block,
+QCryptoBlockReadFunc readfunc,
+QCryptoBlockWriteFunc writefunc,
+void *opaque,
+QCryptoBlockAmendOptions *options,
+bool force,
+Error **errp);
+
 
 /**
  * 

[PATCH v4 02/14] qcrypto/luks: implement encryption key management

[Maxim Levitsky]

Next few patches will expose that functionality
to the user.

Signed-off-by: Maxim Levitsky 
---
 crypto/block-luks.c | 395 +++-
 qapi/crypto.json|  61 ++-
 2 files changed, 452 insertions(+), 4 deletions(-)

diff --git a/crypto/block-luks.c b/crypto/block-luks.c
index 4861db810c..c108518df1 100644
--- a/crypto/block-luks.c
+++ b/crypto/block-luks.c
@@ -32,6 +32,7 @@
 #include "qemu/uuid.h"
 
 #include "qemu/coroutine.h"
+#include "qemu/bitmap.h"
 
 /*
  * Reference for the LUKS format implemented here is
@@ -70,6 +71,9 @@ typedef struct QCryptoBlockLUKSKeySlot 
QCryptoBlockLUKSKeySlot;
 
 #define QCRYPTO_BLOCK_LUKS_SECTOR_SIZE 512LL
 
+#define QCRYPTO_BLOCK_LUKS_DEFAULT_ITER_TIME_MS 2000
+#define QCRYPTO_BLOCK_LUKS_ERASE_ITERATIONS 40
+
 static const char qcrypto_block_luks_magic[QCRYPTO_BLOCK_LUKS_MAGIC_LEN] = {
 'L', 'U', 'K', 'S', 0xBA, 0xBE
 };
@@ -219,6 +223,9 @@ struct QCryptoBlockLUKS {
 
 /* Hash algorithm used in pbkdf2 function */
 QCryptoHashAlgorithm hash_alg;
+
+/* Name of the secret that was used to open the image */
+char *secret;
 };
 
 
@@ -1069,6 +1076,119 @@ qcrypto_block_luks_find_key(QCryptoBlock *block,
 return -1;
 }
 
+/*
+ * Returns true if a slot i is marked as active
+ * (contains encrypted copy of the master key)
+ */
+static bool
+qcrypto_block_luks_slot_active(const QCryptoBlockLUKS *luks,
+   unsigned int slot_idx)
+{
+uint32_t val = luks->header.key_slots[slot_idx].active;
+return val ==  QCRYPTO_BLOCK_LUKS_KEY_SLOT_ENABLED;
+}
+
+/*
+ * Returns the number of slots that are marked as active
+ * (slots that contain encrypted copy of the master key)
+ */
+static unsigned int
+qcrypto_block_luks_count_active_slots(const QCryptoBlockLUKS *luks)
+{
+size_t i = 0;
+unsigned int ret = 0;
+
+for (i = 0; i < QCRYPTO_BLOCK_LUKS_NUM_KEY_SLOTS; i++) {
+if (qcrypto_block_luks_slot_active(luks, i)) {
+ret++;
+}
+}
+return ret;
+}
+
+/*
+ * Finds first key slot which is not active
+ * Returns the key slot index, or -1 if it doesn't exist
+ */
+static int
+qcrypto_block_luks_find_free_keyslot(const QCryptoBlockLUKS *luks)
+{
+size_t i;
+
+for (i = 0; i < QCRYPTO_BLOCK_LUKS_NUM_KEY_SLOTS; i++) {
+if (!qcrypto_block_luks_slot_active(luks, i)) {
+return i;
+}
+}
+return -1;
+}
+
+/*
+ * Erases an keyslot given its index
+ * Returns:
+ *0 if the keyslot was erased successfully
+ *   -1 if a error occurred while erasing the keyslot
+ *
+ */
+static int
+qcrypto_block_luks_erase_key(QCryptoBlock *block,
+ unsigned int slot_idx,
+ QCryptoBlockWriteFunc writefunc,
+ void *opaque,
+ Error **errp)
+{
+QCryptoBlockLUKS *luks = block->opaque;
+QCryptoBlockLUKSKeySlot *slot = >header.key_slots[slot_idx];
+g_autofree uint8_t *garbagesplitkey = NULL;
+size_t splitkeylen = luks->header.master_key_len * slot->stripes;
+size_t i;
+Error *local_err = NULL;
+int ret;
+
+assert(slot_idx < QCRYPTO_BLOCK_LUKS_NUM_KEY_SLOTS);
+assert(splitkeylen > 0);
+garbagesplitkey = g_new0(uint8_t, splitkeylen);
+
+/* Reset the key slot header */
+memset(slot->salt, 0, QCRYPTO_BLOCK_LUKS_SALT_LEN);
+slot->iterations = 0;
+slot->active = QCRYPTO_BLOCK_LUKS_KEY_SLOT_DISABLED;
+
+ret = qcrypto_block_luks_store_header(block,  writefunc,
+  opaque, _err);
+
+if (ret) {
+error_propagate(errp, local_err);
+}
+/*
+ * Now try to erase the key material, even if the header
+ * update failed
+ */
+for (i = 0; i < QCRYPTO_BLOCK_LUKS_ERASE_ITERATIONS; i++) {
+if (qcrypto_random_bytes(garbagesplitkey,
+ splitkeylen, _err) < 0) {
+/*
+ * If we failed to get the random data, still write
+ * at least zeros to the key slot at least once
+ */
+error_propagate(errp, local_err);
+
+if (i > 0) {
+return -1;
+}
+}
+if (writefunc(block,
+  slot->key_offset_sector * QCRYPTO_BLOCK_LUKS_SECTOR_SIZE,
+  garbagesplitkey,
+  splitkeylen,
+  opaque,
+  _err) != splitkeylen) {
+error_propagate(errp, local_err);
+return -1;
+}
+}
+return 0;
+}
 
 static int
 qcrypto_block_luks_open(QCryptoBlock *block,
@@ -1099,6 +1219,7 @@ qcrypto_block_luks_open(QCryptoBlock *block,
 
 luks = g_new0(QCryptoBlockLUKS, 1);
 block->opaque = luks;
+luks->secret = g_strdup(options->u.luks.key_secret);
 
 if (qcrypto_block_luks_load_header(block, readfunc, opaque, errp) < 0) {
 goto fail;
@@ -1164,6 +1285,7 @@ 

Re: [PATCH v1 8/9] hw/dma/xilinx_axidma: s2mm: Support stream fragments

[Alistair Francis]

On Thu, Apr 30, 2020 at 9:31 AM Edgar E. Iglesias
 wrote:
>
> From: "Edgar E. Iglesias" 
>
> Add support for stream fragments.
>
> Signed-off-by: Edgar E. Iglesias 

Reviewed-by: Alistair Francis 

Alistair

> ---
>  hw/dma/xilinx_axidma.c | 14 +++---
>  1 file changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/hw/dma/xilinx_axidma.c b/hw/dma/xilinx_axidma.c
> index 101d32a965..87be9cade7 100644
> --- a/hw/dma/xilinx_axidma.c
> +++ b/hw/dma/xilinx_axidma.c
> @@ -110,6 +110,7 @@ struct Stream {
>
>  int nr;
>
> +bool sof;
>  struct SDesc desc;
>  unsigned int complete_cnt;
>  uint32_t regs[R_MAX];
> @@ -174,6 +175,7 @@ static void stream_reset(struct Stream *s)
>  {
>  s->regs[R_DMASR] = DMASR_HALTED;  /* starts up halted.  */
>  s->regs[R_DMACR] = 1 << 16; /* Starts with one in compl threshold.  */
> +s->sof = true;
>  }
>
>  /* Map an offset addr into a channel index.  */
> @@ -321,12 +323,11 @@ static void stream_process_mem2s(struct Stream *s, 
> StreamSlave *tx_data_dev,
>  }
>
>  static size_t stream_process_s2mem(struct Stream *s, unsigned char *buf,
> -   size_t len)
> +   size_t len, bool eop)
>  {
>  uint32_t prev_d;
>  unsigned int rxlen;
>  size_t pos = 0;
> -int sof = 1;
>
>  if (!stream_running(s) || stream_idle(s)) {
>  return 0;
> @@ -352,16 +353,16 @@ static size_t stream_process_s2mem(struct Stream *s, 
> unsigned char *buf,
>  pos += rxlen;
>
>  /* Update the descriptor.  */
> -if (!len) {
> +if (eop) {
>  stream_complete(s);
>  memcpy(s->desc.app, s->app, sizeof(s->desc.app));
>  s->desc.status |= SDESC_STATUS_EOF;
>  }
>
> -s->desc.status |= sof << SDESC_STATUS_SOF_BIT;
> +s->desc.status |= s->sof << SDESC_STATUS_SOF_BIT;
>  s->desc.status |= SDESC_STATUS_COMPLETE;
>  stream_desc_store(s, s->regs[R_CURDESC]);
> -sof = 0;
> +s->sof = eop;
>
>  /* Advance.  */
>  prev_d = s->regs[R_CURDESC];
> @@ -426,8 +427,7 @@ xilinx_axidma_data_stream_push(StreamSlave *obj, unsigned 
> char *buf, size_t len,
>  struct Stream *s = >dma->streams[1];
>  size_t ret;
>
> -assert(eop);
> -ret = stream_process_s2mem(s, buf, len);
> +ret = stream_process_s2mem(s, buf, len, eop);
>  stream_update_irq(s);
>  return ret;
>  }
> --
> 2.20.1
>
>

[PATCH v4 03/14] block/amend: add 'force' option

[Maxim Levitsky]

'force' option will be used for some unsafe amend operations.

This includes things like erasing last keyslot in luks based formats
(which destroys the data, unless the master key is backed up
by external means), but that _might_ be desired result.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Daniel P. Berrangé 
---
 block.c   | 4 +++-
 block/qcow2.c | 1 +
 docs/tools/qemu-img.rst   | 5 -
 include/block/block.h | 1 +
 include/block/block_int.h | 1 +
 qemu-img-cmds.hx  | 4 ++--
 qemu-img.c| 8 +++-
 7 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index cf5c19b1db..de2e41b361 100644
--- a/block.c
+++ b/block.c
@@ -6377,6 +6377,7 @@ void bdrv_remove_aio_context_notifier(BlockDriverState 
*bs,
 
 int bdrv_amend_options(BlockDriverState *bs, QemuOpts *opts,
BlockDriverAmendStatusCB *status_cb, void *cb_opaque,
+   bool force,
Error **errp)
 {
 if (!bs->drv) {
@@ -6388,7 +6389,8 @@ int bdrv_amend_options(BlockDriverState *bs, QemuOpts 
*opts,
bs->drv->format_name);
 return -ENOTSUP;
 }
-return bs->drv->bdrv_amend_options(bs, opts, status_cb, cb_opaque, errp);
+return bs->drv->bdrv_amend_options(bs, opts, status_cb,
+   cb_opaque, force, errp);
 }
 
 /*
diff --git a/block/qcow2.c b/block/qcow2.c
index 2ba0b17c39..ffb6b22e2d 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -5200,6 +5200,7 @@ static void qcow2_amend_helper_cb(BlockDriverState *bs,
 static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
BlockDriverAmendStatusCB *status_cb,
void *cb_opaque,
+   bool force,
Error **errp)
 {
 BDRVQcow2State *s = bs->opaque;
diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 0080f83a76..fc2dca6649 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -249,11 +249,14 @@ Command description:
 
 .. program:: qemu-img-commands
 
-.. option:: amend [--object OBJECTDEF] [--image-opts] [-p] [-q] [-f FMT] [-t 
CACHE] -o OPTIONS FILENAME
+.. option:: amend [--object OBJECTDEF] [--image-opts] [-p] [-q] [-f FMT] [-t 
CACHE] [--force] -o OPTIONS FILENAME
 
   Amends the image format specific *OPTIONS* for the image file
   *FILENAME*. Not all file formats support this operation.
 
+  --force allows some unsafe operations. Currently for -f luks, it allows to
+  erase last encryption key, and to overwrite an active encryption key.
+
 .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] 
[--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] 
[--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] 
FILENAME
 
   Run a simple sequential I/O benchmark on the specified image. If ``-w`` is
diff --git a/include/block/block.h b/include/block/block.h
index 8b62429aa4..0ca53b5598 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -392,6 +392,7 @@ typedef void BlockDriverAmendStatusCB(BlockDriverState *bs, 
int64_t offset,
   int64_t total_work_size, void *opaque);
 int bdrv_amend_options(BlockDriverState *bs_new, QemuOpts *opts,
BlockDriverAmendStatusCB *status_cb, void *cb_opaque,
+   bool force,
Error **errp);
 
 /* check if a named node can be replaced when doing drive-mirror */
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 92335f33c7..98671ecdf6 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -432,6 +432,7 @@ struct BlockDriver {
 int (*bdrv_amend_options)(BlockDriverState *bs, QemuOpts *opts,
   BlockDriverAmendStatusCB *status_cb,
   void *cb_opaque,
+  bool force,
   Error **errp);
 
 void (*bdrv_debug_event)(BlockDriverState *bs, BlkdebugEvent event);
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index c9c54de1df..9920f1f9d4 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -10,9 +10,9 @@ HXCOMM When amending the rST sections, please remember to 
copy the usage
 HXCOMM over to the per-command sections in qemu-img.texi.
 
 DEF("amend", img_amend,
-"amend [--object objectdef] [--image-opts] [-p] [-q] [-f fmt] [-t cache] 
-o options filename")
+"amend [--object objectdef] [--image-opts] [-p] [-q] [-f fmt] [-t cache] 
[--force] -o options filename")
 SRST
-.. option:: amend [--object OBJECTDEF] [--image-opts] [-p] [-q] [-f FMT] [-t 
CACHE] -o OPTIONS FILENAME
+.. option:: amend [--object OBJECTDEF] [--image-opts] [-p] [-q] [-f FMT] [-t 
CACHE] [--force] -o OPTIONS FILENAME
 ERST
 
 DEF("bench", img_bench,
diff --git a/qemu-img.c b/qemu-img.c
index 6a4327aaba..ef422d5471 100644
--- 

[PATCH v4 00/14] LUKS: encryption slot management using amend interface

[Maxim Levitsky]

Hi!
Here is the updated series of my patches, incorporating all the feedback I 
received.

This implements the API interface that we agreed upon except that I merged the
LUKSKeyslotActive/LUKSKeyslotInactive union into a struct because otherwise
I need nested unions which are not supported currently by QAPI parser.
This didn't change the API and thus once support for nested unions is there,
it can always be implemented in backward compatible way.

I hope that this series will finally be considered for merging, since I am 
somewhat running
out of time to finish this task.

Patches are strictly divided by topic to 3 groups, and each group depends on 
former groups.

* Patches 1,2 implement qcrypto generic amend interface, including definition
  of structs used in crypto.json and implement this in luks crypto driver
  Nothing is exposed to the user at this stage

* Patches 3-9 use the code from patches 1,2 to implement qemu-img amend based 
encryption slot management
  for luks and for qcow2, and add a bunch of iotests to cover that.

* Patches 10-13 add x-blockdev-amend (I'll drop the -x prefix if you like), and 
wire it
  to luks and qcow2 driver to implement qmp based encryption slot management 
also using
  the code from patches 1,2, and also add a bunch of iotests to cover this.

Tested with -raw,-qcow2,-nbd and -luks iotests and 'make check'

Changes from V3: reworked patch #2 to hopefully be more readable and user 
friendly

Best regards,
Maxim Levitsky

clone of "luks-keymgmnt-v2"

Maxim Levitsky (14):
  qcrypto/core: add generic infrastructure for crypto options amendment
  qcrypto/luks: implement encryption key management
  block/amend: add 'force' option
  block/amend: separate amend and create options for qemu-img
  block/amend: refactor qcow2 amend options
  block/crypto: rename two functions
  block/crypto: implement the encryption key management
  block/qcow2: extend qemu-img amend interface with crypto options
  iotests: filter few more luks specific create options
  iotests: qemu-img tests for luks key management
  block/core: add generic infrastructure for x-blockdev-amend qmp
command
  block/crypto: implement blockdev-amend
  block/qcow2: implement blockdev-amend
  iotests: add tests for blockdev-amend

 block.c  |   4 +-
 block/Makefile.objs  |   2 +-
 block/amend.c| 108 +
 block/crypto.c   | 203 ++--
 block/crypto.h   |  37 +++
 block/qcow2.c| 306 ++--
 crypto/block-luks.c  | 395 ++-
 crypto/block.c   |  29 +++
 crypto/blockpriv.h   |   8 +
 docs/tools/qemu-img.rst  |   5 +-
 include/block/block.h|   1 +
 include/block/block_int.h|  24 +-
 include/crypto/block.h   |  22 ++
 qapi/block-core.json |  68 ++
 qapi/crypto.json |  75 +-
 qapi/job.json|   4 +-
 qemu-img-cmds.hx |   4 +-
 qemu-img.c   |  44 +++-
 tests/qemu-iotests/049.out   | 102 
 tests/qemu-iotests/061.out   |  12 +-
 tests/qemu-iotests/079.out   |  18 +-
 tests/qemu-iotests/082.out   | 176 --
 tests/qemu-iotests/085.out   |  38 +--
 tests/qemu-iotests/087.out   |   6 +-
 tests/qemu-iotests/115.out   |   2 +-
 tests/qemu-iotests/121.out   |   4 +-
 tests/qemu-iotests/125.out   | 192 +++
 tests/qemu-iotests/134.out   |   2 +-
 tests/qemu-iotests/144.out   |   4 +-
 tests/qemu-iotests/158.out   |   4 +-
 tests/qemu-iotests/182.out   |   2 +-
 tests/qemu-iotests/185.out   |   8 +-
 tests/qemu-iotests/188.out   |   2 +-
 tests/qemu-iotests/189.out   |   4 +-
 tests/qemu-iotests/198.out   |   4 +-
 tests/qemu-iotests/243.out   |  16 +-
 tests/qemu-iotests/250.out   |   2 +-
 tests/qemu-iotests/255.out   |   8 +-
 tests/qemu-iotests/259.out   |   2 +-
 tests/qemu-iotests/263.out   |   4 +-
 tests/qemu-iotests/274.out   |  46 ++--
 tests/qemu-iotests/280.out   |   2 +-
 tests/qemu-iotests/284.out   |   6 +-
 tests/qemu-iotests/300   | 207 
 tests/qemu-iotests/300.out   |  99 
 tests/qemu-iotests/301   |  90 +++
 tests/qemu-iotests/301.out   |  30 +++
 tests/qemu-iotests/302   | 278 ++
 tests/qemu-iotests/302.out   |  40 
 tests/qemu-iotests/303   | 233 ++
 tests/qemu-iotests/303.out   |  33 +++
 tests/qemu-iotests/common.filter |   6 +-
 tests/qemu-iotests/group |   5 +
 53 files changed, 2493 insertions(+), 533 deletions(-)
 create mode 100644 block/amend.c
 create mode 100755 tests/qemu-iotests/300
 create mode 100644 tests/qemu-iotests/300.out
 create mode 100755 tests/qemu-iotests/301
 create mode 100644 

Re: [PATCH v1 7/9] hw/dma/xilinx_axidma: mm2s: Stream descriptor by descriptor

[Alistair Francis]

On Thu, Apr 30, 2020 at 9:31 AM Edgar E. Iglesias
 wrote:
>
> From: "Edgar E. Iglesias" 
>
> Stream descriptor by descriptor from memory instead of
> buffering entire packets before pushing. This enables
> non-packet streaming clients to work and also lifts the
> limitation that our internal DMA buffer needs to be able
> to hold entire packets.
>
> Signed-off-by: Edgar E. Iglesias 

Reviewed-by: Alistair Francis 

Alistair

> ---
>  hw/dma/xilinx_axidma.c | 31 +--
>  1 file changed, 17 insertions(+), 14 deletions(-)
>
> diff --git a/hw/dma/xilinx_axidma.c b/hw/dma/xilinx_axidma.c
> index a770e12c96..101d32a965 100644
> --- a/hw/dma/xilinx_axidma.c
> +++ b/hw/dma/xilinx_axidma.c
> @@ -111,7 +111,6 @@ struct Stream {
>  int nr;
>
>  struct SDesc desc;
> -int pos;
>  unsigned int complete_cnt;
>  uint32_t regs[R_MAX];
>  uint8_t app[20];
> @@ -267,7 +266,9 @@ static void stream_process_mem2s(struct Stream *s, 
> StreamSlave *tx_data_dev,
>   StreamSlave *tx_control_dev)
>  {
>  uint32_t prev_d;
> -unsigned int txlen;
> +uint32_t txlen;
> +uint64_t addr;
> +bool eop;
>
>  if (!stream_running(s) || stream_idle(s)) {
>  return;
> @@ -282,24 +283,26 @@ static void stream_process_mem2s(struct Stream *s, 
> StreamSlave *tx_data_dev,
>  }
>
>  if (stream_desc_sof(>desc)) {
> -s->pos = 0;
>  stream_push(tx_control_dev, s->desc.app, sizeof(s->desc.app), 
> true);
>  }
>
>  txlen = s->desc.control & SDESC_CTRL_LEN_MASK;
> -if ((txlen + s->pos) > sizeof s->txbuf) {
> -hw_error("%s: too small internal txbuf! %d\n", __func__,
> - txlen + s->pos);
> -}
>
> -address_space_read(>dma->as, s->desc.buffer_address,
> -   MEMTXATTRS_UNSPECIFIED,
> -   s->txbuf + s->pos, txlen);
> -s->pos += txlen;
> +eop = stream_desc_eof(>desc);
> +addr = s->desc.buffer_address;
> +while (txlen) {
> +unsigned int len;
> +
> +len = txlen > sizeof s->txbuf ? sizeof s->txbuf : txlen;
> +address_space_read(>dma->as, addr,
> +   MEMTXATTRS_UNSPECIFIED,
> +   s->txbuf, len);
> +stream_push(tx_data_dev, s->txbuf, len, eop && len == txlen);
> +txlen -= len;
> +addr += len;
> +}
>
> -if (stream_desc_eof(>desc)) {
> -stream_push(tx_data_dev, s->txbuf, s->pos, true);
> -s->pos = 0;
> +if (eop) {
>  stream_complete(s);
>  }
>
> --
> 2.20.1
>
>

Re: [PATCH v1 9/9] MAINTAINERS: Add myself as streams maintainer

[Alistair Francis]

On Thu, Apr 30, 2020 at 9:34 AM Edgar E. Iglesias
 wrote:
>
> From: "Edgar E. Iglesias" 
>
> Since we're missing a maintainer, add myself.
>
> Signed-off-by: Edgar E. Iglesias 

Reviewed-by: Alistair Francis 

Alistair

> ---
>  MAINTAINERS | 6 ++
>  1 file changed, 6 insertions(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8cbc1fac2b..9f504e32df 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2311,6 +2311,12 @@ F: net/slirp.c
>  F: include/net/slirp.h
>  T: git https://people.debian.org/~sthibault/qemu.git slirp
>
> +Streams
> +M: Edgar E. Iglesias 
> +S: Maintained
> +F: hw/core/stream.c
> +F: include/hw/stream.h
> +
>  Stubs
>  M: Paolo Bonzini 
>  S: Maintained
> --
> 2.20.1
>
>

Re: [PATCH v5 13/31] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

Extended L2 entries are 128-bit wide: 64 bits for the entry itself and
64 bits for the subcluster allocation bitmap.

In order to support them correctly get/set_l2_entry() need to be
updated so they take the entry width into account in order to
calculate the correct offset.

This patch also adds the get/set_l2_bitmap() functions that are
used to access the bitmaps. For convenience we allow calling
get_l2_bitmap() on images without subclusters. In this case the
returned value is always 0 and has no meaning.

Signed-off-by: Alberto Garcia 
---
  block/qcow2.h | 21 +
  1 file changed, 21 insertions(+)




+static inline void set_l2_bitmap(BDRVQcow2State *s, uint64_t *l2_slice,
+ int idx, uint64_t bitmap)
+{
+assert(has_subclusters(s));
+idx *= l2_entry_size(s) / sizeof(uint64_t);
+l2_slice[idx + 1] = cpu_to_be64(bitmap);
+}


Unrelated to this patch, but I just thought of it:

What happens for an image whose size is not cluster-aligned?  Must the 
bits corresponding to subclusters not present in the final cluster 
always be zero, or are they instead ignored regardless of value?


But for this patch:

Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH 1/1] target/riscv: fix VS interrupts forwarding to HS

[Alistair Francis]

On Fri, May 1, 2020 at 11:57 AM Jose Martins  wrote:
>
> Reached out to Andrew Waterman. This was his response:
>
> "I think the encoding of the privileged modes is a red herring.  HS is
> inherently more privileged than VS, since it controls memory
> protection and interrupt delegation for VS.
> Certainly the intent is that HS-mode interrupts are always enabled
> while executing in VS-mode.  Otherwise, badly behaved VS-mode software
> could starve HS-mode of interrupts."

Ok, so in which case the hs_sie variable should be removed.

Alistair

>
> So my assumption was correct.
>
> Jose
>
> On Thu, 30 Apr 2020 at 22:47, Jose Martins  wrote:
> >
> > > I'm not sure HS is a higher privilege mode.
> > >
> > > HS is privilege encoding 1, which is the same as VS (VU is obviously 
> > > lower).
> >
> > I just checked the spec and it doesn't actually, explicitly state that
> > HS is a higher-privilege mode than VS. I thought this was something
> > implicit, but you might be right. I'll try to reach out to the spec
> > authors to clarify this.
> >
> > Jose

Re: [PATCH v5 11/31] qcow2: Add offset_into_subcluster() and size_to_subclusters()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

Like offset_into_cluster() and size_to_clusters(), but for
subclusters.

Signed-off-by: Alberto Garcia 
---
  block/qcow2.h | 10 ++
  1 file changed, 10 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h
index e68febb15b..8b1ed1cbcf 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -537,11 +537,21 @@ static inline int64_t offset_into_cluster(BDRVQcow2State 
*s, int64_t offset)
  return offset & (s->cluster_size - 1);
  }
  
+static inline int64_t offset_into_subcluster(BDRVQcow2State *s, int64_t offset)

+{
+return offset & (s->subcluster_size - 1);
+}
+
  static inline uint64_t size_to_clusters(BDRVQcow2State *s, uint64_t size)
  {
  return (size + (s->cluster_size - 1)) >> s->cluster_bits;
  }


Pre-existing, but this could use DIV_ROUND_UP.

  
+static inline uint64_t size_to_subclusters(BDRVQcow2State *s, uint64_t size)

+{
+return (size + (s->subcluster_size - 1)) >> s->subcluster_bits;
+}


at which point, your addition could be:

return DIV_ROUND_UP(size, s->subcluster_size);

Either way,

Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v5 07/31] qcow2: Document the Extended L2 Entries feature

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

Subcluster allocation in qcow2 is implemented by extending the
existing L2 table entries and adding additional information to
indicate the allocation status of each subcluster.

This patch documents the changes to the qcow2 format and how they
affect the calculation of the L2 cache size.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
  docs/interop/qcow2.txt | 68 --
  docs/qcow2-cache.txt   | 19 +++-
  2 files changed, 83 insertions(+), 4 deletions(-)




@@ -547,7 +557,8 @@ Standard Cluster Descriptor:
  nor is data read from the backing file if the cluster is
  unallocated.
  
-With version 2, this is always 0.

+With version 2 or with extended L2 entries (see the next
+section), this is always 0.


In your cover letter, you said you changed things to tolerate this bit 
being set even with extended L2 entries.  Does this sentence need a tweak?


  
   1 -  8:Reserved (set to 0)
  
@@ -584,6 +595,57 @@ file (except if bit 0 in the Standard Cluster Descriptor is set). If there is

  no backing file or the backing file is smaller than the image, they shall read
  zeros for all parts that are not covered by the backing file.
  
+== Extended L2 Entries ==

+
+An image uses Extended L2 Entries if bit 4 is set on the incompatible_features
+field of the header.
+
+In these images standard data clusters are divided into 32 subclusters of the
+same size. They are contiguous and start from the beginning of the cluster.
+Subclusters can be allocated independently and the L2 entry contains 
information
+indicating the status of each one of them. Compressed data clusters don't have
+subclusters so they are treated the same as in images without this feature.
+
+The size of an extended L2 entry is 128 bits so the number of entries per table
+is calculated using this formula:
+
+l2_entries = (cluster_size / (2 * sizeof(uint64_t)))
+
+The first 64 bits have the same format as the standard L2 table entry described
+in the previous section, with the exception of bit 0 of the standard cluster
+descriptor.


Also this sentence.


+
+The last 64 bits contain a subcluster allocation bitmap with this format:
+
+Subcluster Allocation Bitmap (for standard clusters):
+
+Bit  0 -  31:   Allocation status (one bit per subcluster)


Why two spaces after '-'?  I understand it in situations like '0 -  3' 
in the same list as '16 - 19', to make for a right-justified column, but 
here, everything in the second column is two digits, so the extra 
padding doesn't add anything useful.  Or did you mean to have '64 -  95' 
and '96 - 127', making it obvious that these are the second set of bits 
on top of the existing bits in the first 8 bytes?



+
+1: the subcluster is allocated. In this case the
+   host cluster offset field must contain a valid
+   offset.
+0: the subcluster is not allocated. In this case
+   read requests shall go to the backing file or
+   return zeros if there is no backing file data.
+
+Bits are assigned starting from the least significant
+one (i.e. bit x is used for subcluster x).
+
+32 -  63Subcluster reads as zeros (one bit per subcluster)
+
+1: the subcluster reads as zeros. In this case the
+   allocation status bit must be unset. The host
+   cluster offset field may or may not be set.
+0: no effect.
+
+Bits are assigned starting from the least significant
+one (i.e. bit x is used for subcluster x - 32).


Of course, if you change to 64-95 and 96-127, the two sentences mapping 
bit x to subcluster y need adjusting by 64 as well.



+
+Subcluster Allocation Bitmap (for compressed clusters):
+
+Bit  0 -  63:   Reserved (set to 0)
+Compressed clusters don't have subclusters,
+so this field is not used.


I can live with the wording as-is (since you did call out the "second 64 
bits" or with the adjusted bit numberings.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v5 05/31] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

[Eric Blake]

On 5/5/20 12:38 PM, Alberto Garcia wrote:

When writing to a qcow2 file there are two functions that take a
virtual offset and return a host offset, possibly allocating new
clusters if necessary:

- handle_copied() looks for normal data clusters that are already
  allocated and have a reference count of 1. In those clusters we
  can simply write the data and there is no need to perform any
  copy-on-write.

- handle_alloc() looks for clusters that do need copy-on-write,
  either because they haven't been allocated yet, because their
  reference count is != 1 or because they are ZERO_ALLOC clusters.

The ZERO_ALLOC case is a bit special because those are clusters that
are already allocated and they could perfectly be dealt with in
handle_copied() (as long as copy-on-write is performed when required).

In fact, there is extra code specifically for them in handle_alloc()
that tries to reuse the existing allocation if possible and frees them
otherwise.

This patch changes the handling of ZERO_ALLOC clusters so the
semantics of these two functions are now like this:

- handle_copied() looks for clusters that are already allocated and
  which we can overwrite (NORMAL and ZERO_ALLOC clusters with a
  reference count of 1).

- handle_alloc() looks for clusters for which we need a new
  allocation (all other cases).

One important difference after this change is that clusters found
in handle_copied() may now require copy-on-write, but this will be
necessary anyway once we add support for subclusters.

Signed-off-by: Alberto Garcia 
---
  block/qcow2-cluster.c | 256 +++---
  1 file changed, 141 insertions(+), 115 deletions(-)



@@ -1053,15 +1058,53 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, 
QCowL2Meta *m)
  static void calculate_l2_meta(BlockDriverState *bs,
uint64_t host_cluster_offset,
uint64_t guest_offset, unsigned bytes,
-  QCowL2Meta **m, bool keep_old)
+  uint64_t *l2_slice, QCowL2Meta **m, bool 
keep_old)
  {


Borderline long line, but it fits ;)


  BDRVQcow2State *s = bs->opaque;
-unsigned cow_start_from = 0;
+int l2_index = offset_to_l2_slice_index(s, guest_offset);
+uint64_t l2_entry;
+unsigned cow_start_from, cow_end_to;
  unsigned cow_start_to = offset_into_cluster(s, guest_offset);
  unsigned cow_end_from = cow_start_to + bytes;
-unsigned cow_end_to = ROUND_UP(cow_end_from, s->cluster_size);
  unsigned nb_clusters = size_to_clusters(s, cow_end_from);
  QCowL2Meta *old_m = *m;
+QCow2ClusterType type;
+
+assert(nb_clusters <= s->l2_slice_size - l2_index);
+
+/* Return if there's no COW (all clusters are normal and we keep them) */
+if (keep_old) {
+int i;
+for (i = 0; i < nb_clusters; i++) {
+l2_entry = be64_to_cpu(l2_slice[l2_index + i]);
+if (qcow2_get_cluster_type(bs, l2_entry) != QCOW2_CLUSTER_NORMAL) {
+break;
+}
+}
+if (i == nb_clusters) {
+return;
+}
+}
+
+/* Get the L2 entry of the first cluster */
+l2_entry = be64_to_cpu(l2_slice[l2_index]);


This is the second time we're grabbing the first entry in this function. 
But I don't think it's worth trying to micro-optimize.




+static int count_single_write_clusters(BlockDriverState *bs, int nb_clusters,
+   uint64_t *l2_slice, int l2_index,
+   bool new_alloc)
  {
+BDRVQcow2State *s = bs->opaque;
+uint64_t l2_entry = be64_to_cpu(l2_slice[l2_index]);
+uint64_t expected_offset = l2_entry & L2E_OFFSET_MASK;
  int i;
  
  for (i = 0; i < nb_clusters; i++) {

-uint64_t l2_entry = be64_to_cpu(l2_slice[l2_index + i]);
-if (!cluster_needs_cow(bs, l2_entry)) {
+l2_entry = be64_to_cpu(l2_slice[l2_index + i]);


And another place where we compute l2_entry for the first cluster twice, 
and again not worth micro-optimizing.


I didn't find anything that needs a change.

Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[Bug 1805256] Re: qemu-img hangs on rcu_call_ready_event logic in Aarch64 when converting images

[Rafael David Tinoco]

Hello Ike,

Please, let me know if you want me to go after the needed SRUs for this
fix or if you will.

I'll wait for the final feedback from tests with your PPA.

Cheers!

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1805256

Title:
  qemu-img hangs on rcu_call_ready_event logic in Aarch64 when
  converting images

Status in kunpeng920:
  Triaged
Status in kunpeng920 ubuntu-18.04 series:
  New
Status in kunpeng920 ubuntu-18.04-hwe series:
  New
Status in kunpeng920 ubuntu-19.10 series:
  New
Status in kunpeng920 ubuntu-20.04 series:
  New
Status in kunpeng920 upstream-kernel series:
  Fix Committed
Status in QEMU:
  In Progress
Status in qemu package in Ubuntu:
  Incomplete
Status in qemu source package in Bionic:
  Incomplete
Status in qemu source package in Disco:
  Incomplete
Status in qemu source package in Eoan:
  Incomplete
Status in qemu source package in Focal:
  Incomplete

Bug description:
  Command:

  qemu-img convert -f qcow2 -O qcow2 ./disk01.qcow2 ./output.qcow2

  Hangs indefinitely approximately 30% of the runs.

  

  Workaround:

  qemu-img convert -m 1 -f qcow2 -O qcow2 ./disk01.qcow2 ./output.qcow2

  Run "qemu-img convert" with "a single coroutine" to avoid this issue.

  

  (gdb) thread 1
  ...
  (gdb) bt
  #0 0xbf1ad81c in __GI_ppoll
  #1 0xaabcf73c in ppoll
  #2 qemu_poll_ns
  #3 0xaabd0764 in os_host_main_loop_wait
  #4 main_loop_wait
  ...

  (gdb) thread 2
  ...
  (gdb) bt
  #0 syscall ()
  #1 0xaabd41cc in qemu_futex_wait
  #2 qemu_event_wait (ev=ev@entry=0xaac86ce8 )
  #3 0xaabed05c in call_rcu_thread
  #4 0xaabd34c8 in qemu_thread_start
  #5 0xbf25c880 in start_thread
  #6 0xbf1b6b9c in thread_start ()

  (gdb) thread 3
  ...
  (gdb) bt
  #0 0xbf11aa20 in __GI___sigtimedwait
  #1 0xbf2671b4 in __sigwait
  #2 0xaabd1ddc in sigwait_compat
  #3 0xaabd34c8 in qemu_thread_start
  #4 0xbf25c880 in start_thread
  #5 0xbf1b6b9c in thread_start

  

  (gdb) run
  Starting program: /usr/bin/qemu-img convert -f qcow2 -O qcow2
  ./disk01.ext4.qcow2 ./output.qcow2

  [New Thread 0xbec5ad90 (LWP 72839)]
  [New Thread 0xbe459d90 (LWP 72840)]
  [New Thread 0xbdb57d90 (LWP 72841)]
  [New Thread 0xacac9d90 (LWP 72859)]
  [New Thread 0xa7ffed90 (LWP 72860)]
  [New Thread 0xa77fdd90 (LWP 72861)]
  [New Thread 0xa6ffcd90 (LWP 72862)]
  [New Thread 0xa67fbd90 (LWP 72863)]
  [New Thread 0xa5ffad90 (LWP 72864)]

  [Thread 0xa5ffad90 (LWP 72864) exited]
  [Thread 0xa6ffcd90 (LWP 72862) exited]
  [Thread 0xa77fdd90 (LWP 72861) exited]
  [Thread 0xbdb57d90 (LWP 72841) exited]
  [Thread 0xa67fbd90 (LWP 72863) exited]
  [Thread 0xacac9d90 (LWP 72859) exited]
  [Thread 0xa7ffed90 (LWP 72860) exited]

  
  """

  All the tasks left are blocked in a system call, so no task left to call
  qemu_futex_wake() to unblock thread #2 (in futex()), which would unblock
  thread #1 (doing poll() in a pipe with thread #2).

  Those 7 threads exit before disk conversion is complete (sometimes in
  the beginning, sometimes at the end).

  

  [ Original Description ]

  On the HiSilicon D06 system - a 96 core NUMA arm64 box - qemu-img
  frequently hangs (~50% of the time) with this command:

  qemu-img convert -f qcow2 -O qcow2 /tmp/cloudimg /tmp/cloudimg2

  Where "cloudimg" is a standard qcow2 Ubuntu cloud image. This
  qcow2->qcow2 conversion happens to be something uvtool does every time
  it fetches images.

  Once hung, attaching gdb gives the following backtrace:

  (gdb) bt
  #0  0xae4f8154 in __GI_ppoll (fds=0xe8a67dc0, 
nfds=187650274213760,
  timeout=, timeout@entry=0x0, sigmask=0xc123b950)
  at ../sysdeps/unix/sysv/linux/ppoll.c:39
  #1  0xbbefaf00 in ppoll (__ss=0x0, __timeout=0x0, __nfds=,
  __fds=) at /usr/include/aarch64-linux-gnu/bits/poll2.h:77
  #2  qemu_poll_ns (fds=, nfds=,
  timeout=timeout@entry=-1) at util/qemu-timer.c:322
  #3  0xbbefbf80 in os_host_main_loop_wait (timeout=-1)
  at util/main-loop.c:233
  #4  main_loop_wait (nonblocking=) at util/main-loop.c:497
  #5  0xbbe2aa30 in convert_do_copy (s=0xc123bb58) at 
qemu-img.c:1980
  #6  img_convert (argc=, argv=) at 
qemu-img.c:2456
  #7  0xbbe2333c in main (argc=7, argv=) at 
qemu-img.c:4975

  Reproduced w/ latest QEMU git (@ 53744e0a182)

To manage notifications about this bug go to:
https://bugs.launchpad.net/kunpeng920/+bug/1805256/+subscriptions

[PATCH v2 0/1] target-ppc: fix rlwimi, rlwinm, rlwnm for Clang-9

[Daniele Buono]

This patch fixes a compilation error with Clang v9 and higher in
target/ppc/translate.c, on a comparison that is always true in PPC32
because of type sizes.

More information about the issue are in the first version of the patch.

v2, changed to avoid the nested ifdef/conditional solution of v1, and
keep a code structure more similar to the original.

Daniele Buono (1):
  target-ppc: fix rlwimi, rlwinm, rlwnm for Clang-9

 target/ppc/translate.c | 24 +---
 1 file changed, 21 insertions(+), 3 deletions(-)

-- 
2.26.2

[PATCH v2 1/1] target-ppc: fix rlwimi, rlwinm, rlwnm for Clang-9

[Daniele Buono]

Starting with Clang v9, -Wtype-limits is implemented and triggers a
few "result of comparison is always true" errors when compiling PPC32
targets.

The comparisons seem to be necessary only on PPC64, since the
else branch in PPC32 only has a "g_assert_not_reached();" in all cases.

This patch restructures the code so that the actual if/else is done on a
local flag variable, that is set accordingly for PPC64, and always
true for PPC32.

Signed-off-by: Daniele Buono 
---
 target/ppc/translate.c | 24 +---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/target/ppc/translate.c b/target/ppc/translate.c
index 807d14faaa..338529879f 100644
--- a/target/ppc/translate.c
+++ b/target/ppc/translate.c
@@ -1882,6 +1882,7 @@ static void gen_rlwimi(DisasContext *ctx)
 tcg_gen_deposit_tl(t_ra, t_ra, t_rs, sh, me - mb + 1);
 } else {
 target_ulong mask;
+bool mask_in_32b = true;
 TCGv t1;
 
 #if defined(TARGET_PPC64)
@@ -1890,8 +1891,13 @@ static void gen_rlwimi(DisasContext *ctx)
 #endif
 mask = MASK(mb, me);
 
+#if defined(TARGET_PPC64)
+if (mask > 0xu) {
+mask_in_32b = false;
+}
+#endif
 t1 = tcg_temp_new();
-if (mask <= 0xu) {
+if (mask_in_32b) {
 TCGv_i32 t0 = tcg_temp_new_i32();
 tcg_gen_trunc_tl_i32(t0, t_rs);
 tcg_gen_rotli_i32(t0, t0, sh);
@@ -1933,12 +1939,18 @@ static void gen_rlwinm(DisasContext *ctx)
 tcg_gen_extract_tl(t_ra, t_rs, rsh, len);
 } else {
 target_ulong mask;
+bool mask_in_32b = true;
 #if defined(TARGET_PPC64)
 mb += 32;
 me += 32;
 #endif
 mask = MASK(mb, me);
-if (mask <= 0xu) {
+#if defined(TARGET_PPC64)
+if (mask > 0xu) {
+mask_in_32b = false;
+}
+#endif
+if (mask_in_32b) {
 if (sh == 0) {
 tcg_gen_andi_tl(t_ra, t_rs, mask);
 } else {
@@ -1973,6 +1985,7 @@ static void gen_rlwnm(DisasContext *ctx)
 uint32_t mb = MB(ctx->opcode);
 uint32_t me = ME(ctx->opcode);
 target_ulong mask;
+bool mask_in_32b = true;
 
 #if defined(TARGET_PPC64)
 mb += 32;
@@ -1980,7 +1993,12 @@ static void gen_rlwnm(DisasContext *ctx)
 #endif
 mask = MASK(mb, me);
 
-if (mask <= 0xu) {
+#if defined(TARGET_PPC64)
+if (mask > 0xu) {
+mask_in_32b = false;
+}
+#endif
+if (mask_in_32b) {
 TCGv_i32 t0 = tcg_temp_new_i32();
 TCGv_i32 t1 = tcg_temp_new_i32();
 tcg_gen_trunc_tl_i32(t0, t_rb);
-- 
2.26.2

Re: [PATCH v2 00/18] qom: Spring cleaning

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200505152926.18877-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505152926.18877-1-arm...@redhat.com
Subject: [PATCH v2 00/18] qom: Spring cleaning
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag] 
patchew/20200505174520.cd62a3a0...@moya.office.hostfission.com -> 
patchew/20200505174520.cd62a3a0...@moya.office.hostfission.com
Switched to a new branch 'test'
4cba1a0 qom: Drop @errp parameter of object_property_del()
d8b7296 spapr_pci: Drop some dead error handling
f3700d4 qdev: Unrealize must not fail
f39f651 Drop more @errp parameters after previous commit
93d8e99 qom: Drop parameter @errp of object_property_add() & friends
6a392b4 qdev: Clean up qdev_connect_gpio_out_named()
6f7f8c4 hw/arm/bcm2835: Drop futile attempts at QOM-adopting memory
da916a5 e1000: Don't run e1000_instance_init() twice
f2cc352 hw/isa/superio: Make the components QOM children
98ac27d s390x/cpumodel: Fix UI to CPU features pcc-cmac-{aes, eaes}-256
b88e1ea tests/check-qom-proplist: Improve iterator coverage
065d2f5 qom: Drop object_property_set_description() parameter @errp
eb48860 qom: Make all the object_property_add_FOO() return the property
9647d4a qom: Drop convenience method object_property_get_uint16List()
4f21ae5 qom: Simplify object_property_get_enum()
8bf7167 qom: Drop object_property_del_child()'s unused parameter @errp
c86c469 qom: Clean up inconsistent use of gchar * vs. char *
24a691d qom: Clearer reference counting in object_initialize_childv()

=== OUTPUT BEGIN ===
1/18 Checking commit 24a691def353 (qom: Clearer reference counting in 
object_initialize_childv())
2/18 Checking commit c86c4696bff7 (qom: Clean up inconsistent use of gchar * 
vs. char *)
3/18 Checking commit 8bf71679bce1 (qom: Drop object_property_del_child()'s 
unused parameter @errp)
4/18 Checking commit 4f21ae5ae5ed (qom: Simplify object_property_get_enum())
5/18 Checking commit 9647d4ae16b7 (qom: Drop convenience method 
object_property_get_uint16List())
6/18 Checking commit eb48860a3c26 (qom: Make all the object_property_add_FOO() 
return the property)
7/18 Checking commit 065d2f55d1d4 (qom: Drop object_property_set_description() 
parameter @errp)
8/18 Checking commit b88e1ea4bc39 (tests/check-qom-proplist: Improve iterator 
coverage)
9/18 Checking commit 98ac27d47088 (s390x/cpumodel: Fix UI to CPU features 
pcc-cmac-{aes, eaes}-256)
ERROR: line over 90 characters
#53: FILE: target/s390x/cpu_features_def.inc.h:314:
+DEF_FEAT(PCC_CMAC_AES_256, "pcc-cmac-aes-256", PCC, 20, "PCC 
Compute-Last-Block-CMAC-Using-AES-256")

total: 1 errors, 0 warnings, 8 lines checked

Patch 9/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/18 Checking commit f2cc352a601b (hw/isa/superio: Make the components QOM 
children)
11/18 Checking commit da916a5c390b (e1000: Don't run e1000_instance_init() 
twice)
12/18 Checking commit 6f7f8c446ca6 (hw/arm/bcm2835: Drop futile attempts at 
QOM-adopting memory)
13/18 Checking commit 6a392b4df970 (qdev: Clean up 
qdev_connect_gpio_out_named())
14/18 Checking commit 93d8e9905c38 (qom: Drop parameter @errp of 
object_property_add() & friends)
WARNING: line over 80 characters
#209: FILE: backends/hostmem-file.c:187:
+file_memory_backend_get_discard_data, 
file_memory_backend_set_discard_data);

WARNING: line over 80 characters
#1080: FILE: hw/arm/raspi.c:287:
+object_property_add_const_link(OBJECT(>soc), "ram", 
OBJECT(machine->ram));

WARNING: line over 80 characters
#3095: FILE: hw/ppc/spapr.c:3346:
+   >kernel_addr, 
OBJ_PROP_FLAG_READWRITE);

total: 0 errors, 3 warnings, 4471 lines checked

Patch 14/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/18 Checking commit f39f6513b7a2 (Drop more @errp parameters after previous 
commit)
16/18 Checking commit f3700d44eab1 (qdev: Unrealize must not fail)
17/18 Checking commit d8b7296cf934 (spapr_pci: Drop some dead error handling)
18/18 Checking commit 4cba1a0de99a (qom: Drop @errp parameter of 
object_property_del())
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505152926.18877-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [PATCH v2 0/4] softfloat: fix floatx80 emulation bugs

[Richard Henderson]

On 5/4/20 4:36 PM, Joseph Myers wrote:
> Joseph Myers (4):
>   softfloat: silence sNaN for conversions to/from floatx80
>   softfloat: fix floatx80 pseudo-denormal addition / subtraction
>   softfloat: fix floatx80 pseudo-denormal comparisons
>   softfloat: fix floatx80 pseudo-denormal round to integer

Thanks.  Queued to a local branch with some more fpu cleanups.


r~

Re: Postcopy with different page sizes on source and destination

[Dr. David Alan Gilbert]

* Florian Schmidt (flo...@nutanix.com) wrote:
> Hi,

Hi Florian,

> with precopy live migration, change in page size on source and 
> destination is possible: using hugetlbfs memory backing for the VM on 
> the source and anonymous memory on the destination, and vice versa. For 
> postcopy migration, this is not allowed, and in fact checked during the 
> advise stage.
> 
> Is there any fundamental limitation in the design that prevents this, or 
> is it more that this is an additional complication that nobody has 
> implemented so far because there was no strong need for it?
> 
> It seems to me like this should be possible, and the comment in 
> loadvm_postcopy_handle_advise() (migration/savevm.c:1681) also seems to 
> suggest that; so I'll add a (very rough) first idea.

Yeh it was getting hairy enough at the time, so I kept that restriction.
Now let me just reload that from my brain from 3 years or so back

> Please tell me if 
> I'm missing something important. The "background" copy is similar to 
> precopy, so the main difference is the userfaultfd page fault handling 
> on the destination, and requesting the correct memory from the source.

Right.

> 1. If the source has hugepages and the destination doesn't, then a page 
> fault would lead the destination to ask "I need these 4k of memory from 
> you to fill my page and handle the page fault". The source could then 
> answer "here you are, and here are these other 511 4k pages around it 
> (which form my 2M page; similarly for 1G pages), please deal with them 
> now". That way, even "release-ram" would still work on a (huge)page 
> granularity.

Yeh I think that's about right; you might have to watch out for cases
where the RAMBlock sizes are different because they've got rounded.

> 2. If the destination has hugepages and the source doesn't, then the 
> above works similarly: now the destination, on a page fault, asks for a 
> larger memory area that corresponds to 512 (or more) pages on the 
> source. The only issue I could see here is during the initial phase, 
> when postcopy is switched on, to make sure that the source doesn't 
> release RAM that it has copied and thinks is clean, but it part of a 
> hugepage on the other side. That seems easy enough to solve though? And 
> indeed is probably already implemented for precopy migration to work 
> with different page sizes on source and destination and could be adapted 
> here.

Precopy doesn't have to worry about it because it doesn't have to clear
out previously partially sent pages.

You'd have a situation where the source things page p+8k is dirty so
sends a discard for that; the destination can't do that - so what does
it do?  You need to get the source to discard on the largest granularity
of source and destination.

The other problem you have here is making sure that the source really
does send all the pages continuously starting from the right point so
that they all end up in one chunk on the destination and it can perform
a place;  for example imagine the source is doing a background
page transfer and is currently at x+1MB,  now it gets a request
from the destination for page y, so it switches to transmitting 'y'
which given the destinations request it will probably transfer
the whole of y - but x was partially transmitted which means
x won't have got placed on the destination.  I'd also worry about
whether the code on the source is OK if it gets a request for 'z'
while it's sending y, but it's probably ok because it has the
counter.

(I'm not sure if there are any changes needed in postcopy recovery -
that was more recent).

Note it's not just hugepages either; you get aarch and power systems
that can have configurable base page sizes, but it's rare to mix
them.

Dave

> Cheers,
> Florian
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [PATCH v6] audio/jack: add JACK client audiodev

[no-reply]

Patchew URL: 
https://patchew.org/QEMU/20200505151055.5ead73a0...@moya.office.hostfission.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20200505151055.5ead73a0...@moya.office.hostfission.com
Subject: [PATCH v6] audio/jack: add JACK client audiodev
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
40e14fb audio/jack: add JACK client audiodev

=== OUTPUT BEGIN ===
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#52: 
new file mode 100644

ERROR: "foo * bar" should be "foo *bar"
#716: FILE: audio/jackaudio.c:660:
+static void qjack_error(const char * msg)

ERROR: "foo * bar" should be "foo *bar"
#721: FILE: audio/jackaudio.c:665:
+static void qjack_info(const char * msg)

total: 2 errors, 1 warnings, 806 lines checked

Commit 40e14fbebc24 (audio/jack: add JACK client audiodev) has style problems, 
please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200505151055.5ead73a0...@moya.office.hostfission.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[PATCH v7] audio/jack: add JACK client audiodev

[Geoffrey McRae]

This commit adds a new audiodev backend to allow QEMU to use JACK as
both an audio sink and source.

Signed-off-by: Geoffrey McRae 
---
 audio/Makefile.objs|   5 +
 audio/audio.c  |   1 +
 audio/audio_template.h |   2 +
 audio/jackaudio.c  | 677 +
 configure  |  17 ++
 qapi/audio.json|  56 +++-
 6 files changed, 756 insertions(+), 2 deletions(-)
 create mode 100644 audio/jackaudio.c

diff --git a/audio/Makefile.objs b/audio/Makefile.objs
index d7490a379f..b4a4c11f31 100644
--- a/audio/Makefile.objs
+++ b/audio/Makefile.objs
@@ -28,3 +28,8 @@ common-obj-$(CONFIG_AUDIO_SDL) += sdl.mo
 sdl.mo-objs = sdlaudio.o
 sdl.mo-cflags := $(SDL_CFLAGS)
 sdl.mo-libs := $(SDL_LIBS)
+
+# jack module
+common-obj-$(CONFIG_AUDIO_JACK) += jack.mo
+jack.mo-objs = jackaudio.o
+jack.mo-libs := $(JACK_LIBS)
diff --git a/audio/audio.c b/audio/audio.c
index 7a9e680355..95d9fb16ca 100644
--- a/audio/audio.c
+++ b/audio/audio.c
@@ -1969,6 +1969,7 @@ void audio_create_pdos(Audiodev *dev)
 CASE(ALSA, alsa, Alsa);
 CASE(COREAUDIO, coreaudio, Coreaudio);
 CASE(DSOUND, dsound, );
+CASE(JACK, jack, Jack);
 CASE(OSS, oss, Oss);
 CASE(PA, pa, Pa);
 CASE(SDL, sdl, );
diff --git a/audio/audio_template.h b/audio/audio_template.h
index 7013d3041f..8dd48ce14e 100644
--- a/audio/audio_template.h
+++ b/audio/audio_template.h
@@ -330,6 +330,8 @@ AudiodevPerDirectionOptions *glue(audio_get_pdo_, 
TYPE)(Audiodev *dev)
 dev->u.coreaudio.TYPE);
 case AUDIODEV_DRIVER_DSOUND:
 return dev->u.dsound.TYPE;
+case AUDIODEV_DRIVER_JACK:
+return qapi_AudiodevJackPerDirectionOptions_base(dev->u.jack.TYPE);
 case AUDIODEV_DRIVER_OSS:
 return qapi_AudiodevOssPerDirectionOptions_base(dev->u.oss.TYPE);
 case AUDIODEV_DRIVER_PA:
diff --git a/audio/jackaudio.c b/audio/jackaudio.c
new file mode 100644
index 00..34563f5a13
--- /dev/null
+++ b/audio/jackaudio.c
@@ -0,0 +1,677 @@
+/*
+ * QEMU JACK Audio Connection Kit Client
+ *
+ * Copyright (c) 2020 Geoffrey McRae (gnif)
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/module.h"
+#include "qemu/fifo8.h"
+#include "qemu-common.h"
+#include "audio.h"
+
+#define AUDIO_CAP "jack"
+#include "audio_int.h"
+
+#include 
+#include 
+#include 
+
+struct QJack;
+
+typedef enum QJackState {
+QJACK_STATE_DISCONNECTED,
+QJACK_STATE_STOPPED,
+QJACK_STATE_RUNNING,
+QJACK_STATE_SHUTDOWN
+}
+QJackState;
+
+typedef struct QJackBuffer {
+int  channels;
+int  frames;
+_Atomic(int) used;
+int  rptr, wptr;
+float  **data;
+}
+QJackBuffer;
+
+typedef struct QJackClient {
+AudiodevJackPerDirectionOptions *opt;
+
+bool out;
+bool finished;
+bool connect_ports;
+int  packets;
+
+QJackState  state;
+jack_client_t  *client;
+jack_nframes_t  freq;
+
+struct QJack   *j;
+int nchannels;
+int buffersize;
+jack_port_t   **port;
+QJackBuffer fifo;
+}
+QJackClient;
+
+typedef struct QJackOut {
+HWVoiceOut  hw;
+QJackClient c;
+}
+QJackOut;
+
+typedef struct QJackIn {
+HWVoiceIn   hw;
+QJackClient c;
+}
+QJackIn;
+
+static int qjack_client_init(QJackClient *c);
+static void qjack_client_connect_ports(QJackClient *c);
+static void qjack_client_fini(QJackClient *c);
+
+static void qjack_buffer_create(QJackBuffer *buffer, int channels, int frames)
+{
+buffer->channels = channels;
+buffer->frames   = frames;
+buffer->used = 0;
+buffer->rptr = 0;
+buffer->wptr = 0;
+buffer->data = g_malloc(channels * sizeof(float *));
+for (int i = 0; i < channels; ++i) {
+buffer->data[i] = g_malloc(frames * sizeof(float));
+}
+}
+
+static void qjack_buffer_clear(QJackBuffer *buffer)
+{
+

[PATCH v5 16/31] qcow2: Add cluster type parameter to qcow2_get_host_offset()

[Alberto Garcia]

This function returns an integer that can be either an error code or a
cluster type (a value from the QCow2ClusterType enum).

We are going to start using subcluster types instead of cluster types
in some functions so it's better to use the exact data types instead
of integers for clarity and in order to detect errors more easily.

This patch makes qcow2_get_host_offset() return 0 on success and
puts the returned cluster type in a separate parameter. There are no
semantic changes.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.h |  3 ++-
 block/qcow2-cluster.c | 11 +++
 block/qcow2.c | 37 ++---
 3 files changed, 31 insertions(+), 20 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index b5db8d2f36..cca650e934 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -876,7 +876,8 @@ int qcow2_encrypt_sectors(BDRVQcow2State *s, int64_t 
sector_num,
   uint8_t *buf, int nb_sectors, bool enc, Error 
**errp);
 
 int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
-  unsigned int *bytes, uint64_t *host_offset);
+  unsigned int *bytes, uint64_t *host_offset,
+  QCow2ClusterType *cluster_type);
 int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
unsigned int *bytes, uint64_t *host_offset,
QCowL2Meta **m);
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 8b2fc550b7..64481067ce 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -514,13 +514,14 @@ static int coroutine_fn 
do_perform_cow_write(BlockDriverState *bs,
  *
  * On exit, *bytes is the number of bytes starting at offset that have the same
  * cluster type and (if applicable) are stored contiguously in the image file.
+ * The cluster type is stored in *cluster_type.
  * Compressed clusters are always returned one by one.
  *
- * Returns the cluster type (QCOW2_CLUSTER_*) on success, -errno in error
- * cases.
+ * Returns 0 on success, -errno in error cases.
  */
 int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
-  unsigned int *bytes, uint64_t *host_offset)
+  unsigned int *bytes, uint64_t *host_offset,
+  QCow2ClusterType *cluster_type)
 {
 BDRVQcow2State *s = bs->opaque;
 unsigned int l2_index;
@@ -661,7 +662,9 @@ out:
 assert(bytes_available - offset_in_cluster <= UINT_MAX);
 *bytes = bytes_available - offset_in_cluster;
 
-return type;
+*cluster_type = type;
+
+return 0;
 
 fail:
 qcow2_cache_put(s->l2_table_cache, (void **)_slice);
diff --git a/block/qcow2.c b/block/qcow2.c
index 9a8fb5a7bf..b9effb195b 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1973,6 +1973,7 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 BDRVQcow2State *s = bs->opaque;
 uint64_t host_offset;
 unsigned int bytes;
+QCow2ClusterType type;
 int ret, status = 0;
 
 qemu_co_mutex_lock(>lock);
@@ -1984,7 +1985,7 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 }
 
 bytes = MIN(INT_MAX, count);
-ret = qcow2_get_host_offset(bs, offset, , _offset);
+ret = qcow2_get_host_offset(bs, offset, , _offset, );
 qemu_co_mutex_unlock(>lock);
 if (ret < 0) {
 return ret;
@@ -1992,15 +1993,15 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 
 *pnum = bytes;
 
-if ((ret == QCOW2_CLUSTER_NORMAL || ret == QCOW2_CLUSTER_ZERO_ALLOC) &&
+if ((type == QCOW2_CLUSTER_NORMAL || type == QCOW2_CLUSTER_ZERO_ALLOC) &&
 !s->crypto) {
 *map = host_offset;
 *file = s->data_file->bs;
 status |= BDRV_BLOCK_OFFSET_VALID;
 }
-if (ret == QCOW2_CLUSTER_ZERO_PLAIN || ret == QCOW2_CLUSTER_ZERO_ALLOC) {
+if (type == QCOW2_CLUSTER_ZERO_PLAIN || type == QCOW2_CLUSTER_ZERO_ALLOC) {
 status |= BDRV_BLOCK_ZERO;
-} else if (ret != QCOW2_CLUSTER_UNALLOCATED) {
+} else if (type != QCOW2_CLUSTER_UNALLOCATED) {
 status |= BDRV_BLOCK_DATA;
 }
 if (s->metadata_preallocation && (status & BDRV_BLOCK_DATA) &&
@@ -2209,6 +2210,7 @@ static coroutine_fn int 
qcow2_co_preadv_part(BlockDriverState *bs,
 int ret = 0;
 unsigned int cur_bytes; /* number of bytes in current iteration */
 uint64_t host_offset = 0;
+QCow2ClusterType type;
 AioTaskPool *aio = NULL;
 
 while (bytes != 0 && aio_task_pool_status(aio) == 0) {
@@ -2220,22 +,23 @@ static coroutine_fn int 
qcow2_co_preadv_part(BlockDriverState *bs,
 }
 
 qemu_co_mutex_lock(>lock);
-ret = qcow2_get_host_offset(bs, offset, _bytes, _offset);
+ret = qcow2_get_host_offset(bs, offset, _bytes,
+_offset, );
 

[PATCH v5 25/31] qcow2: Clear the L2 bitmap when allocating a compressed cluster

[Alberto Garcia]

Compressed clusters always have the bitmap part of the extended L2
entry set to 0.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
 block/qcow2-cluster.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 4544a40aa0..0a295076a3 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -800,6 +800,9 @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState 
*bs,
 BLKDBG_EVENT(bs->file, BLKDBG_L2_UPDATE_COMPRESSED);
 qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice);
 set_l2_entry(s, l2_slice, l2_index, cluster_offset);
+if (has_subclusters(s)) {
+set_l2_bitmap(s, l2_slice, l2_index, 0);
+}
 qcow2_cache_put(s->l2_table_cache, (void **) _slice);
 
 *host_offset = cluster_offset & s->cluster_offset_mask;
-- 
2.20.1

[PATCH v5 23/31] qcow2: Add subcluster support to check_refcounts_l2()

[Alberto Garcia]

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an
image has subclusters. Instead, the individual 'all zeroes' bits must
be used.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-refcount.c | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index dfdcdd3c25..9bb161481e 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1686,8 +1686,13 @@ static int check_refcounts_l2(BlockDriverState *bs, 
BdrvCheckResult *res,
 int ign = active ? QCOW2_OL_ACTIVE_L2 :
QCOW2_OL_INACTIVE_L2;
 
-l2_entry = QCOW_OFLAG_ZERO;
-set_l2_entry(s, l2_table, i, l2_entry);
+if (has_subclusters(s)) {
+set_l2_entry(s, l2_table, i, 0);
+set_l2_bitmap(s, l2_table, i,
+  QCOW_L2_BITMAP_ALL_ZEROES);
+} else {
+set_l2_entry(s, l2_table, i, QCOW_OFLAG_ZERO);
+}
 ret = qcow2_pre_write_overlap_check(bs, ign,
 l2e_offset, l2_entry_size(s), false);
 if (ret < 0) {
-- 
2.20.1

[PATCH v5 26/31] qcow2: Add subcluster support to handle_alloc_space()

[Alberto Garcia]

The bdrv_co_pwrite_zeroes() call here fills complete clusters with
zeroes, but it can happen that some subclusters are not part of the
write request or the copy-on-write. This patch makes sure that only
the affected subclusters are overwritten.

A potential improvement would be to also fill with zeroes the other
subclusters if we can guarantee that we are not overwriting existing
data. However this would waste more disk space, so we should first
evaluate if it's really worth doing.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.c | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 63e952b89a..cc5591fe8f 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2351,6 +2351,9 @@ static int handle_alloc_space(BlockDriverState *bs, 
QCowL2Meta *l2meta)
 
 for (m = l2meta; m != NULL; m = m->next) {
 int ret;
+uint64_t start_offset = m->alloc_offset + m->cow_start.offset;
+unsigned nb_bytes = m->cow_end.offset + m->cow_end.nb_bytes -
+m->cow_start.offset;
 
 if (!m->cow_start.nb_bytes && !m->cow_end.nb_bytes) {
 continue;
@@ -2365,16 +2368,14 @@ static int handle_alloc_space(BlockDriverState *bs, 
QCowL2Meta *l2meta)
  * efficiently zero out the whole clusters
  */
 
-ret = qcow2_pre_write_overlap_check(bs, 0, m->alloc_offset,
-m->nb_clusters * s->cluster_size,
+ret = qcow2_pre_write_overlap_check(bs, 0, start_offset, nb_bytes,
 true);
 if (ret < 0) {
 return ret;
 }
 
 BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_SPACE);
-ret = bdrv_co_pwrite_zeroes(s->data_file, m->alloc_offset,
-m->nb_clusters * s->cluster_size,
+ret = bdrv_co_pwrite_zeroes(s->data_file, start_offset, nb_bytes,
 BDRV_REQ_NO_FALLBACK);
 if (ret < 0) {
 if (ret != -ENOTSUP && ret != -EAGAIN) {
-- 
2.20.1

[PATCH v5 00/31] Add subcluster allocation to qcow2

[Alberto Garcia]

Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Important changes here:

- I fixed hopefully all of the issues mentioned in the previous
  review. Thanks to everyone who contributed.

- There's now support for partial zeroing of clusters (i.e. at the
  subcluster level).

- Many more tests.

- QCOW_OFLAG_ZERO is simply ignored now and not considered a sign of a
  corrupt image anymore. I hesitated about this, but we could still
  add that check later. I think there's a case for adding a new
  QCOW2_CLUSTER_INVALID type and include this and other scenarios that
  we already consider corrupt (for example: clusters with unaligned
  offsets). We would need to see if for 'qemu-img check' adding
  QCOW2_CLUSTER_INVALID complicates things or not. But I think that
  all is material for its own series.

And I think that's all. See below for the detailed list of changes,
and thanks again for the feedback.

Berto

v5:
- Patch 01: Fix indentation [Max], add trace event [Vladimir]
- Patch 02: Add host_cluster_offset variable [Vladirmir]
- Patch 05: Have separate l2_entry and cluster_offset variables [Vladimir]
- Patch 06: Only context changes due to patch 05
- Patch 11: New patch
- Patch 13: Change documentation of get_l2_entry()
- Patch 14: Add QCOW_OFLAG_SUB_{ALLOC,ZERO}_RANGE [Eric] and rewrite
the other macros.
Ignore QCOW_OFLAG_ZERO on images with subclusters
(i.e. don't treat them as corrupted).
- Patch 15: New patch
- Patch 19: Optimize cow by skipping all leading and trailing zero and
unallocated subclusters [Vladimir]
Return 0 on success [Vladimir]
Squash patch that updated handle_dependencies() [Vladirmir]
- Patch 20: Call count_contiguous_subclusters() after the main switch
in qcow2_get_host_offset() [Vladimir]
Add assertion and remove goto statement [Vladimir]
- Patch 21: Rewrite algorithm.
- Patch 22: Rewrite algorithm.
- Patch 24: Replace loop with the _RANGE macros from patch 14 [Eric]
- Patch 27: New patch
- Patch 28: Update version number and expected output from tests.
- Patch 31: Add many more new tests

v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html
v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v4:

Key:
[] : patches are identical
[] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/31:[0005] [FC] 'qcow2: Make Qcow2AioTask store the full host offset'
002/31:[0018] [FC] 'qcow2: Convert qcow2_get_cluster_offset() into 
qcow2_get_host_offset()'
003/31:[] [--] 'qcow2: Add calculate_l2_meta()'
004/31:[] [--] 'qcow2: Split cluster_needs_cow() out of 
count_cow_clusters()'
005/31:[0038] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in 
handle_copied()'
006/31:[0004] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/31:[] [--] 'qcow2: Document the Extended L2 Entries feature'
008/31:[] [--] 'qcow2: Add dummy has_subclusters() function'
009/31:[] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/31:[] [--] 'qcow2: Add offset_to_sc_index()'
011/31:[down] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()'
012/31:[] [--] 'qcow2: Add l2_entry_size()'
013/31:[0003] [FC] 'qcow2: Update get/set_l2_entry() and add 
get/set_l2_bitmap()'
014/31:[0023] [FC] 'qcow2: Add QCow2SubclusterType and 
qcow2_get_subcluster_type()'
015/31:[down] 'qcow2: Add qcow2_cluster_is_allocated()'
016/31:[] [--] 'qcow2: Add cluster type parameter to 
qcow2_get_host_offset()'
017/31:[] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
018/31:[] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
019/31:[0066] [FC] 'qcow2: Add subcluster support to calculate_l2_meta()'
020/31:[0022] [FC] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
021/31:[0040] [FC] 'qcow2: Add subcluster support to zero_in_l2_slice()'
022/31:[0061] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()'
023/31:[] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
024/31:[0019] [FC] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
025/31:[] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed 
cluster'
026/31:[] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
027/31:[down] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()'
028/31:[0105] [FC] 'qcow2: Add the 'extended_l2' option and the 
QCOW2_INCOMPAT_EXTL2 bit'

[PATCH v5 19/31] qcow2: Add subcluster support to calculate_l2_meta()

[Alberto Garcia]

If an image has subclusters then there are more copy-on-write
scenarios that we need to consider. Let's say we have a write request
from the middle of subcluster #3 until the end of the cluster:

1) If we are writing to a newly allocated cluster then we need
   copy-on-write. The previous contents of subclusters #0 to #3 must
   be copied to the new cluster. We can optimize this process by
   skipping all leading unallocated or zero subclusters (the status of
   those skipped subclusters will be reflected in the new L2 bitmap).

2) If we are overwriting an existing cluster:

   2.1) If subcluster #3 is unallocated or has the all-zeroes bit set
then we need copy-on-write (on subcluster #3 only).

   2.2) If subcluster #3 was already allocated then there is no need
for any copy-on-write. However we still need to update the L2
bitmap to reflect possible changes in the allocation status of
subclusters #4 to #31. Because of this, this function checks
if all the overwritten subclusters are already allocated and
in this case it returns without creating a new QCowL2Meta
structure.

After all these changes l2meta_cow_start() and l2meta_cow_end()
are not necessarily cluster-aligned anymore. We need to update the
calculation of old_start and old_end in handle_dependencies() to
guarantee that no two requests try to write on the same cluster.

Signed-off-by: Alberto Garcia 
---
 block/qcow2-cluster.c | 174 +++---
 1 file changed, 146 insertions(+), 28 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 5595ce1404..ffcb11edda 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1059,56 +1059,156 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, 
QCowL2Meta *m)
  * If @keep_old is true it means that the clusters were already
  * allocated and will be overwritten. If false then the clusters are
  * new and we have to decrease the reference count of the old ones.
+ *
+ * Returns 0 on success, -errno on failure.
  */
-static void calculate_l2_meta(BlockDriverState *bs,
-  uint64_t host_cluster_offset,
-  uint64_t guest_offset, unsigned bytes,
-  uint64_t *l2_slice, QCowL2Meta **m, bool 
keep_old)
+static int calculate_l2_meta(BlockDriverState *bs, uint64_t 
host_cluster_offset,
+ uint64_t guest_offset, unsigned bytes,
+ uint64_t *l2_slice, QCowL2Meta **m, bool keep_old)
 {
 BDRVQcow2State *s = bs->opaque;
-int l2_index = offset_to_l2_slice_index(s, guest_offset);
-uint64_t l2_entry;
+int sc_index, l2_index = offset_to_l2_slice_index(s, guest_offset);
+uint64_t l2_entry, l2_bitmap;
 unsigned cow_start_from, cow_end_to;
 unsigned cow_start_to = offset_into_cluster(s, guest_offset);
 unsigned cow_end_from = cow_start_to + bytes;
 unsigned nb_clusters = size_to_clusters(s, cow_end_from);
 QCowL2Meta *old_m = *m;
-QCow2ClusterType type;
+QCow2SubclusterType type;
 
 assert(nb_clusters <= s->l2_slice_size - l2_index);
 
-/* Return if there's no COW (all clusters are normal and we keep them) */
+/* Return if there's no COW (all subclusters are normal and we are
+ * keeping the clusters) */
 if (keep_old) {
+unsigned first_sc = cow_start_to / s->subcluster_size;
+unsigned last_sc = (cow_end_from - 1) / s->subcluster_size;
 int i;
-for (i = 0; i < nb_clusters; i++) {
-l2_entry = get_l2_entry(s, l2_slice, l2_index + i);
-if (qcow2_get_cluster_type(bs, l2_entry) != QCOW2_CLUSTER_NORMAL) {
+for (i = first_sc; i <= last_sc; i++) {
+unsigned c = i / s->subclusters_per_cluster;
+unsigned sc = i % s->subclusters_per_cluster;
+l2_entry = get_l2_entry(s, l2_slice, l2_index + c);
+l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index + c);
+type = qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, sc);
+if (type == QCOW2_SUBCLUSTER_INVALID) {
+l2_index += c; /* Point to the invalid entry */
+goto fail;
+}
+if (type != QCOW2_SUBCLUSTER_NORMAL) {
 break;
 }
 }
-if (i == nb_clusters) {
-return;
+if (i == last_sc + 1) {
+return 0;
 }
 }
 
 /* Get the L2 entry of the first cluster */
 l2_entry = get_l2_entry(s, l2_slice, l2_index);
-type = qcow2_get_cluster_type(bs, l2_entry);
+l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index);
+sc_index = offset_to_sc_index(s, guest_offset);
+type = qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, sc_index);
 
-if (type == QCOW2_CLUSTER_NORMAL && keep_old) {
-cow_start_from = cow_start_to;
+if (type == QCOW2_SUBCLUSTER_INVALID) {
+goto fail;
+}
+
+

[PATCH v5 30/31] qcow2: Add subcluster support to qcow2_measure()

[Alberto Garcia]

Extended L2 entries are bigger than normal L2 entries so this has an
impact on the amount of metadata needed for a qcow2 file.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
 block/qcow2.c | 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 18c8e3f52a..31d72f1297 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3173,28 +3173,31 @@ int64_t qcow2_refcount_metadata_size(int64_t clusters, 
size_t cluster_size,
  * @total_size: virtual disk size in bytes
  * @cluster_size: cluster size in bytes
  * @refcount_order: refcount bits power-of-2 exponent
+ * @extended_l2: true if the image has extended L2 entries
  *
  * Returns: Total number of bytes required for the fully allocated image
  * (including metadata).
  */
 static int64_t qcow2_calc_prealloc_size(int64_t total_size,
 size_t cluster_size,
-int refcount_order)
+int refcount_order,
+bool extended_l2)
 {
 int64_t meta_size = 0;
 uint64_t nl1e, nl2e;
 int64_t aligned_total_size = ROUND_UP(total_size, cluster_size);
+size_t l2e_size = extended_l2 ? L2E_SIZE_EXTENDED : L2E_SIZE_NORMAL;
 
 /* header: 1 cluster */
 meta_size += cluster_size;
 
 /* total size of L2 tables */
 nl2e = aligned_total_size / cluster_size;
-nl2e = ROUND_UP(nl2e, cluster_size / sizeof(uint64_t));
-meta_size += nl2e * sizeof(uint64_t);
+nl2e = ROUND_UP(nl2e, cluster_size / l2e_size);
+meta_size += nl2e * l2e_size;
 
 /* total size of L1 tables */
-nl1e = nl2e * sizeof(uint64_t) / cluster_size;
+nl1e = nl2e * l2e_size / cluster_size;
 nl1e = ROUND_UP(nl1e, cluster_size / sizeof(uint64_t));
 meta_size += nl1e * sizeof(uint64_t);
 
@@ -4765,6 +4768,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, 
BlockDriverState *in_bs,
 bool has_backing_file;
 bool has_luks;
 bool extended_l2;
+size_t l2e_size;
 
 /* Parse image creation options */
 extended_l2 = qemu_opt_get_bool_del(opts, BLOCK_OPT_EXTL2, false);
@@ -4833,8 +4837,9 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, 
BlockDriverState *in_bs,
 virtual_size = ROUND_UP(virtual_size, cluster_size);
 
 /* Check that virtual disk size is valid */
+l2e_size = extended_l2 ? L2E_SIZE_EXTENDED : L2E_SIZE_NORMAL;
 l2_tables = DIV_ROUND_UP(virtual_size / cluster_size,
- cluster_size / sizeof(uint64_t));
+ cluster_size / l2e_size);
 if (l2_tables * sizeof(uint64_t) > QCOW_MAX_L1_SIZE) {
 error_setg(_err, "The image size is too large "
"(try using a larger cluster size)");
@@ -4897,9 +4902,9 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, 
BlockDriverState *in_bs,
 }
 
 info = g_new(BlockMeasureInfo, 1);
-info->fully_allocated =
+info->fully_allocated = luks_payload_size +
 qcow2_calc_prealloc_size(virtual_size, cluster_size,
- ctz32(refcount_bits)) + luks_payload_size;
+ ctz32(refcount_bits), extended_l2);
 
 /* Remove data clusters that are not required.  This overestimates the
  * required size because metadata needed for the fully allocated file is
-- 
2.20.1

[PATCH v5 31/31] iotests: Add tests for qcow2 images with extended L2 entries

[Alberto Garcia]

Signed-off-by: Alberto Garcia 
---
 tests/qemu-iotests/271 | 664 +
 tests/qemu-iotests/271.out | 519 +
 tests/qemu-iotests/group   |   1 +
 3 files changed, 1184 insertions(+)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271
new file mode 100755
index 00..2df0dac00f
--- /dev/null
+++ b/tests/qemu-iotests/271
@@ -0,0 +1,664 @@
+#!/bin/bash
+#
+# Test qcow2 images with extended L2 entries
+#
+# Copyright (C) 2019-2020 Igalia, S.L.
+# Author: Alberto Garcia 
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=be...@igalia.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+rm -f "$TEST_IMG.raw"
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file nfs
+_supported_os Linux
+_unsupported_imgopts extended_l2 compat=0.10 cluster_size data_file
+
+l2_offset=262144 # 0x4
+
+_verify_img()
+{
+$QEMU_IMG compare "$TEST_IMG" "$TEST_IMG.raw" | grep -v 'Images are 
identical'
+$QEMU_IMG check "$TEST_IMG" | _filter_qemu_img_check | \
+grep -v 'No errors were found on the image'
+}
+
+# Compare the bitmap of an extended L2 entry against an expected value
+_verify_l2_bitmap()
+{
+entry_no="$1"# L2 entry number, starting from 0
+expected_alloc="$2"  # Space-separated list of allocated subcluster indexes
+expected_zero="$3"   # Space-separated list of zero subcluster indexes
+
+offset=$(($l2_offset + $entry_no * 16))
+entry=`peek_file_be "$TEST_IMG" $offset 8`
+offset=$(($offset + 8))
+bitmap=`peek_file_be "$TEST_IMG" $offset 8`
+
+expected_bitmap=0
+for bit in $expected_alloc; do
+expected_bitmap=$(($expected_bitmap | (1 << $bit)))
+done
+for bit in $expected_zero; do
+expected_bitmap=$(($expected_bitmap | (1 << (32 + $bit
+done
+expected_bitmap=`printf "%llu" $expected_bitmap`
+
+printf "L2 entry #%d: 0x%016lx %016lx\n" "$entry_no" "$entry" "$bitmap"
+if [ "$bitmap" != "$expected_bitmap" ]; then
+printf "ERROR: expecting bitmap   0x%016lx\n" "$expected_bitmap"
+fi
+}
+
+_test_write()
+{
+cmd="$1"
+alloc_bitmap="$2"
+zero_bitmap="$3"
+l2_entry_idx="$4"
+[ -n "$l2_entry_idx" ] || l2_entry_idx=0
+raw_cmd=`echo $cmd | sed s/-c//` # Raw images don't support -c
+echo "$cmd"
+$QEMU_IO -c "$cmd" "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "$raw_cmd" -f raw "$TEST_IMG.raw" | _filter_qemu_io
+_verify_img
+_verify_l2_bitmap "$l2_entry_idx" "$alloc_bitmap" "$zero_bitmap"
+}
+
+_reset_img()
+{
+size="$1"
+$QEMU_IMG create -f raw "$TEST_IMG.raw" "$size" | _filter_img_create
+if [ "$use_backing_file" = "yes" ]; then
+$QEMU_IMG create -f raw "$TEST_IMG.base" "$size" | _filter_img_create
+$QEMU_IO -c "write -q -P 0xFF 0 $size" -f raw "$TEST_IMG.base" | 
_filter_qemu_io
+$QEMU_IO -c "write -q -P 0xFF 0 $size" -f raw "$TEST_IMG.raw" | 
_filter_qemu_io
+_make_test_img -o extended_l2=on -b "$TEST_IMG.base" "$size"
+else
+_make_test_img -o extended_l2=on "$size"
+fi
+}
+
+# Test that writing to an image with subclusters produces the expected
+# results, in images with and without backing files
+for use_backing_file in yes no; do
+echo
+echo "### Standard write tests (backing file: $use_backing_file) ###"
+echo
+_reset_img 1M
+### Write subcluster #0 (beginning of subcluster) ###
+alloc="0"; zero=""
+_test_write 'write -q -P 1 0 1k' "$alloc" "$zero"
+
+### Write subcluster #1 (middle of subcluster) ###
+alloc="0 1"; zero=""
+_test_write 'write -q -P 2 3k 512' "$alloc" "$zero"
+
+### Write subcluster #2 (end of subcluster) ###
+alloc="0 1 2"; zero=""
+_test_write 'write -q -P 3 5k 1k' "$alloc" "$zero"
+
+### Write subcluster #3 (full subcluster) ###
+alloc="0 1 2 3"; zero=""
+_test_write 'write -q -P 4 6k 2k' "$alloc" "$zero"
+
+### Write subclusters #4-6 (full subclusters) ###
+alloc="`seq 0 6`"; zero=""
+_test_write 'write -q 

[PATCH v5 17/31] qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*

[Alberto Garcia]

In order to support extended L2 entries some functions of the qcow2
driver need to start dealing with subclusters instead of clusters.

qcow2_get_host_offset() is modified to return the subcluster type
instead of the cluster type, and all callers are updated to replace
all values of QCow2ClusterType with their QCow2SubclusterType
equivalents.

This patch only changes the data types, there are no semantic changes.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.h |  2 +-
 block/qcow2-cluster.c | 10 +++
 block/qcow2.c | 70 ++-
 3 files changed, 42 insertions(+), 40 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index cca650e934..1663b5359c 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -877,7 +877,7 @@ int qcow2_encrypt_sectors(BDRVQcow2State *s, int64_t 
sector_num,
 
 int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
   unsigned int *bytes, uint64_t *host_offset,
-  QCow2ClusterType *cluster_type);
+  QCow2SubclusterType *subcluster_type);
 int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
unsigned int *bytes, uint64_t *host_offset,
QCowL2Meta **m);
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 64481067ce..5595ce1404 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -513,15 +513,15 @@ static int coroutine_fn 
do_perform_cow_write(BlockDriverState *bs,
  * offset that we are interested in.
  *
  * On exit, *bytes is the number of bytes starting at offset that have the same
- * cluster type and (if applicable) are stored contiguously in the image file.
- * The cluster type is stored in *cluster_type.
- * Compressed clusters are always returned one by one.
+ * subcluster type and (if applicable) are stored contiguously in the image
+ * file. The subcluster type is stored in *subcluster_type.
+ * Compressed clusters are always processed one by one.
  *
  * Returns 0 on success, -errno in error cases.
  */
 int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
   unsigned int *bytes, uint64_t *host_offset,
-  QCow2ClusterType *cluster_type)
+  QCow2SubclusterType *subcluster_type)
 {
 BDRVQcow2State *s = bs->opaque;
 unsigned int l2_index;
@@ -662,7 +662,7 @@ out:
 assert(bytes_available - offset_in_cluster <= UINT_MAX);
 *bytes = bytes_available - offset_in_cluster;
 
-*cluster_type = type;
+*subcluster_type = qcow2_cluster_to_subcluster_type(type);
 
 return 0;
 
diff --git a/block/qcow2.c b/block/qcow2.c
index b9effb195b..13965f2e1d 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1973,7 +1973,7 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 BDRVQcow2State *s = bs->opaque;
 uint64_t host_offset;
 unsigned int bytes;
-QCow2ClusterType type;
+QCow2SubclusterType type;
 int ret, status = 0;
 
 qemu_co_mutex_lock(>lock);
@@ -1993,15 +1993,16 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 
 *pnum = bytes;
 
-if ((type == QCOW2_CLUSTER_NORMAL || type == QCOW2_CLUSTER_ZERO_ALLOC) &&
-!s->crypto) {
+if ((type == QCOW2_SUBCLUSTER_NORMAL ||
+ type == QCOW2_SUBCLUSTER_ZERO_ALLOC) && !s->crypto) {
 *map = host_offset;
 *file = s->data_file->bs;
 status |= BDRV_BLOCK_OFFSET_VALID;
 }
-if (type == QCOW2_CLUSTER_ZERO_PLAIN || type == QCOW2_CLUSTER_ZERO_ALLOC) {
+if (type == QCOW2_SUBCLUSTER_ZERO_PLAIN ||
+type == QCOW2_SUBCLUSTER_ZERO_ALLOC) {
 status |= BDRV_BLOCK_ZERO;
-} else if (type != QCOW2_CLUSTER_UNALLOCATED) {
+} else if (type != QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN) {
 status |= BDRV_BLOCK_DATA;
 }
 if (s->metadata_preallocation && (status & BDRV_BLOCK_DATA) &&
@@ -2098,7 +2099,7 @@ typedef struct Qcow2AioTask {
 AioTask task;
 
 BlockDriverState *bs;
-QCow2ClusterType cluster_type; /* only for read */
+QCow2SubclusterType subcluster_type; /* only for read */
 uint64_t host_offset; /* or full descriptor in compressed clusters */
 uint64_t offset;
 uint64_t bytes;
@@ -2111,7 +2112,7 @@ static coroutine_fn int 
qcow2_co_preadv_task_entry(AioTask *task);
 static coroutine_fn int qcow2_add_task(BlockDriverState *bs,
AioTaskPool *pool,
AioTaskFunc func,
-   QCow2ClusterType cluster_type,
+   QCow2SubclusterType subcluster_type,
uint64_t host_offset,
uint64_t offset,
uint64_t bytes,
@@ -2125,7 

[PATCH v5 05/31] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

[Alberto Garcia]

When writing to a qcow2 file there are two functions that take a
virtual offset and return a host offset, possibly allocating new
clusters if necessary:

   - handle_copied() looks for normal data clusters that are already
 allocated and have a reference count of 1. In those clusters we
 can simply write the data and there is no need to perform any
 copy-on-write.

   - handle_alloc() looks for clusters that do need copy-on-write,
 either because they haven't been allocated yet, because their
 reference count is != 1 or because they are ZERO_ALLOC clusters.

The ZERO_ALLOC case is a bit special because those are clusters that
are already allocated and they could perfectly be dealt with in
handle_copied() (as long as copy-on-write is performed when required).

In fact, there is extra code specifically for them in handle_alloc()
that tries to reuse the existing allocation if possible and frees them
otherwise.

This patch changes the handling of ZERO_ALLOC clusters so the
semantics of these two functions are now like this:

   - handle_copied() looks for clusters that are already allocated and
 which we can overwrite (NORMAL and ZERO_ALLOC clusters with a
 reference count of 1).

   - handle_alloc() looks for clusters for which we need a new
 allocation (all other cases).

One important difference after this change is that clusters found
in handle_copied() may now require copy-on-write, but this will be
necessary anyway once we add support for subclusters.

Signed-off-by: Alberto Garcia 
---
 block/qcow2-cluster.c | 256 +++---
 1 file changed, 141 insertions(+), 115 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 80f9787461..fce0be7a08 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1039,13 +1039,18 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, 
QCowL2Meta *m)
 
 /*
  * For a given write request, create a new QCowL2Meta structure, add
- * it to @m and the BDRVQcow2State.cluster_allocs list.
+ * it to @m and the BDRVQcow2State.cluster_allocs list. If the write
+ * request does not need copy-on-write or changes to the L2 metadata
+ * then this function does nothing.
  *
  * @host_cluster_offset points to the beginning of the first cluster.
  *
  * @guest_offset and @bytes indicate the offset and length of the
  * request.
  *
+ * @l2_slice contains the L2 entries of all clusters involved in this
+ * write request.
+ *
  * If @keep_old is true it means that the clusters were already
  * allocated and will be overwritten. If false then the clusters are
  * new and we have to decrease the reference count of the old ones.
@@ -1053,15 +1058,53 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, 
QCowL2Meta *m)
 static void calculate_l2_meta(BlockDriverState *bs,
   uint64_t host_cluster_offset,
   uint64_t guest_offset, unsigned bytes,
-  QCowL2Meta **m, bool keep_old)
+  uint64_t *l2_slice, QCowL2Meta **m, bool 
keep_old)
 {
 BDRVQcow2State *s = bs->opaque;
-unsigned cow_start_from = 0;
+int l2_index = offset_to_l2_slice_index(s, guest_offset);
+uint64_t l2_entry;
+unsigned cow_start_from, cow_end_to;
 unsigned cow_start_to = offset_into_cluster(s, guest_offset);
 unsigned cow_end_from = cow_start_to + bytes;
-unsigned cow_end_to = ROUND_UP(cow_end_from, s->cluster_size);
 unsigned nb_clusters = size_to_clusters(s, cow_end_from);
 QCowL2Meta *old_m = *m;
+QCow2ClusterType type;
+
+assert(nb_clusters <= s->l2_slice_size - l2_index);
+
+/* Return if there's no COW (all clusters are normal and we keep them) */
+if (keep_old) {
+int i;
+for (i = 0; i < nb_clusters; i++) {
+l2_entry = be64_to_cpu(l2_slice[l2_index + i]);
+if (qcow2_get_cluster_type(bs, l2_entry) != QCOW2_CLUSTER_NORMAL) {
+break;
+}
+}
+if (i == nb_clusters) {
+return;
+}
+}
+
+/* Get the L2 entry of the first cluster */
+l2_entry = be64_to_cpu(l2_slice[l2_index]);
+type = qcow2_get_cluster_type(bs, l2_entry);
+
+if (type == QCOW2_CLUSTER_NORMAL && keep_old) {
+cow_start_from = cow_start_to;
+} else {
+cow_start_from = 0;
+}
+
+/* Get the L2 entry of the last cluster */
+l2_entry = be64_to_cpu(l2_slice[l2_index + nb_clusters - 1]);
+type = qcow2_get_cluster_type(bs, l2_entry);
+
+if (type == QCOW2_CLUSTER_NORMAL && keep_old) {
+cow_end_to = cow_end_from;
+} else {
+cow_end_to = ROUND_UP(cow_end_from, s->cluster_size);
+}
 
 *m = g_malloc0(sizeof(**m));
 **m = (QCowL2Meta) {
@@ -1087,18 +1130,22 @@ static void calculate_l2_meta(BlockDriverState *bs,
 QLIST_INSERT_HEAD(>cluster_allocs, *m, next_in_flight);
 }
 
-/* Returns true if writing to a cluster requires COW 

[PATCH v5 28/31] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

[Alberto Garcia]

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with this feature,
which we call "extended L2 entries".

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
 qapi/block-core.json |   7 +++
 block/qcow2.h|   8 ++-
 include/block/block_int.h|   1 +
 block/qcow2.c|  65 ++--
 tests/qemu-iotests/031.out   |   8 +--
 tests/qemu-iotests/036.out   |   4 +-
 tests/qemu-iotests/049.out   | 102 +++
 tests/qemu-iotests/060.out   |   1 +
 tests/qemu-iotests/061.out   |  20 +++---
 tests/qemu-iotests/065   |  18 --
 tests/qemu-iotests/082.out   |  48 ---
 tests/qemu-iotests/085.out   |  38 ++--
 tests/qemu-iotests/144.out   |   4 +-
 tests/qemu-iotests/182.out   |   2 +-
 tests/qemu-iotests/185.out   |   8 +--
 tests/qemu-iotests/198.out   |   2 +
 tests/qemu-iotests/206.out   |   4 ++
 tests/qemu-iotests/242.out   |   5 ++
 tests/qemu-iotests/255.out   |   8 +--
 tests/qemu-iotests/274.out   |  49 ---
 tests/qemu-iotests/280.out   |   2 +-
 tests/qemu-iotests/common.filter |   1 +
 22 files changed, 266 insertions(+), 139 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 943df1926a..f5d1656001 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -66,6 +66,9 @@
 # standalone (read-only) raw image without looking at qcow2
 # metadata (since: 4.0)
 #
+# @extended-l2: true if the image has extended L2 entries; only valid for
+#   compat >= 1.1 (since 5.1)
+#
 # @lazy-refcounts: on or off; only valid for compat >= 1.1
 #
 # @corrupt: true if the image has been marked corrupt; only valid for
@@ -85,6 +88,7 @@
   'compat': 'str',
   '*data-file': 'str',
   '*data-file-raw': 'bool',
+  '*extended-l2': 'bool',
   '*lazy-refcounts': 'bool',
   '*corrupt': 'bool',
   'refcount-bits': 'int',
@@ -4296,6 +4300,8 @@
 # @data-file-raw: True if the external data file must stay valid as a
 # standalone (read-only) raw image without looking at qcow2
 # metadata (default: false; since: 4.0)
+# @extended-l2  True to make the image have extended L2 entries
+#   (default: false; since 5.1)
 # @size: Size of the virtual disk in bytes
 # @version: Compatibility level (default: v3)
 # @backing-file: File name of the backing file if a backing file
@@ -4314,6 +4320,7 @@
   'data': { 'file': 'BlockdevRef',
 '*data-file':   'BlockdevRef',
 '*data-file-raw':   'bool',
+'*extended-l2': 'bool',
 'size': 'size',
 '*version': 'BlockdevQcow2Version',
 '*backing-file':'str',
diff --git a/block/qcow2.h b/block/qcow2.h
index 7349c6ce40..09bf561305 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -237,13 +237,16 @@ enum {
 QCOW2_INCOMPAT_DIRTY_BITNR  = 0,
 QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
 QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
+QCOW2_INCOMPAT_EXTL2_BITNR  = 4,
 QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
 QCOW2_INCOMPAT_CORRUPT  = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
 QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+QCOW2_INCOMPAT_EXTL2= 1 << QCOW2_INCOMPAT_EXTL2_BITNR,
 
 QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
 | QCOW2_INCOMPAT_CORRUPT
-| QCOW2_INCOMPAT_DATA_FILE,
+| QCOW2_INCOMPAT_DATA_FILE
+| QCOW2_INCOMPAT_EXTL2,
 };
 
 /* Compatible feature bits */
@@ -555,8 +558,7 @@ typedef enum QCow2MetadataOverlap {
 
 static inline bool has_subclusters(BDRVQcow2State *s)
 {
-/* FIXME: Return false until this feature is complete */
-return false;
+return s->incompatible_features & QCOW2_INCOMPAT_EXTL2;
 }
 
 static inline size_t l2_entry_size(BDRVQcow2State *s)
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 92335f33c7..27f17bab78 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
 #define BLOCK_OPT_DATA_FILE "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
+#define BLOCK_OPT_EXTL2 "extended_l2"
 
 #define BLOCK_PROBE_BUF_SIZE512
 
diff --git a/block/qcow2.c b/block/qcow2.c
index ec4d1405f0..18c8e3f52a 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1385,6 +1385,12 @@ static int coroutine_fn qcow2_do_open(BlockDriverState 
*bs, QDict *options,
 s->subcluster_size = s->cluster_size / s->subclusters_per_cluster;
 s->subcluster_bits = ctz32(s->subcluster_size);
 

[PATCH v5 14/31] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

[Alberto Garcia]

This patch adds QCow2SubclusterType, which is the subcluster-level
version of QCow2ClusterType. All QCOW2_SUBCLUSTER_* values have the
the same meaning as their QCOW2_CLUSTER_* equivalents (when they
exist). See below for details and caveats.

In images without extended L2 entries clusters are treated as having
exactly one subcluster so it is possible to replace one data type with
the other while keeping the exact same semantics.

With extended L2 entries there are new possible values, and every
subcluster in the same cluster can obviously have a different
QCow2SubclusterType so functions need to be adapted to work on the
subcluster level.

There are several things that have to be taken into account:

  a) QCOW2_SUBCLUSTER_COMPRESSED means that the whole cluster is
 compressed. We do not support compression at the subcluster
 level.

  b) There are two different values for unallocated subclusters:
 QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN which means that the whole
 cluster is unallocated, and QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
 which means that the cluster is allocated but the subcluster is
 not. The latter can only happen in images with extended L2
 entries.

  c) QCOW2_SUBCLUSTER_INVALID is used to detect the cases where an L2
 entry has a value that violates the specification. The caller is
 responsible for handling these situations.

 To prevent compatibility problems with images that have invalid
 values but are currently being read by QEMU without causing side
 effects, QCOW2_SUBCLUSTER_INVALID is only returned for images
 with extended L2 entries.

qcow2_cluster_to_subcluster_type() is added as a separate function
from qcow2_get_subcluster_type(), but this is only temporary and both
will be merged in a subsequent patch.

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h | 127 +-
 1 file changed, 126 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 4ad93772b9..be7816a3b8 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -80,6 +80,21 @@
 
 #define QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER 32
 
+/* The subcluster X [0..31] is allocated */
+#define QCOW_OFLAG_SUB_ALLOC(X)   (1ULL << (X))
+/* The subcluster X [0..31] reads as zeroes */
+#define QCOW_OFLAG_SUB_ZERO(X)(QCOW_OFLAG_SUB_ALLOC(X) << 32)
+/* Subclusters X to Y (both included) are allocated */
+#define QCOW_OFLAG_SUB_ALLOC_RANGE(X, Y) \
+(QCOW_OFLAG_SUB_ALLOC((Y) + 1) - QCOW_OFLAG_SUB_ALLOC(X))
+/* Subclusters X to Y (both included) read as zeroes */
+#define QCOW_OFLAG_SUB_ZERO_RANGE(X, Y) \
+(QCOW_OFLAG_SUB_ALLOC_RANGE(X, Y) << 32)
+/* L2 entry bitmap with all allocation bits set */
+#define QCOW_L2_BITMAP_ALL_ALLOC  (QCOW_OFLAG_SUB_ALLOC_RANGE(0, 31))
+/* L2 entry bitmap with all "read as zeroes" bits set */
+#define QCOW_L2_BITMAP_ALL_ZEROES (QCOW_OFLAG_SUB_ZERO_RANGE(0, 31))
+
 /* Size of normal and extended L2 entries */
 #define L2E_SIZE_NORMAL   (sizeof(uint64_t))
 #define L2E_SIZE_EXTENDED (sizeof(uint64_t) * 2)
@@ -444,6 +459,33 @@ typedef struct QCowL2Meta
 QLIST_ENTRY(QCowL2Meta) next_in_flight;
 } QCowL2Meta;
 
+/*
+ * In images with standard L2 entries all clusters are treated as if
+ * they had one subcluster so QCow2ClusterType and QCow2SubclusterType
+ * can be mapped to each other and have the exact same meaning
+ * (QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC cannot happen in these images).
+ *
+ * In images with extended L2 entries QCow2ClusterType refers to the
+ * complete cluster and QCow2SubclusterType to each of the individual
+ * subclusters, so there are several possible combinations:
+ *
+ * |--+---|
+ * | Cluster type | Possible subcluster types |
+ * |--+---|
+ * | UNALLOCATED  | UNALLOCATED_PLAIN |
+ * |  |ZERO_PLAIN |
+ * |--+---|
+ * | NORMAL   | UNALLOCATED_ALLOC |
+ * |  |ZERO_ALLOC |
+ * |  |NORMAL |
+ * |--+---|
+ * | COMPRESSED   |COMPRESSED |
+ * |--+---|
+ *
+ * QCOW2_SUBCLUSTER_INVALID means that the L2 entry is incorrect and
+ * the image should be marked corrupt.
+ */
+
 typedef enum QCow2ClusterType {
 QCOW2_CLUSTER_UNALLOCATED,
 QCOW2_CLUSTER_ZERO_PLAIN,
@@ -452,6 +494,16 @@ typedef enum QCow2ClusterType {
 QCOW2_CLUSTER_COMPRESSED,
 } QCow2ClusterType;
 
+typedef enum QCow2SubclusterType {
+QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN,
+QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC,
+QCOW2_SUBCLUSTER_ZERO_PLAIN,
+QCOW2_SUBCLUSTER_ZERO_ALLOC,
+QCOW2_SUBCLUSTER_NORMAL,
+QCOW2_SUBCLUSTER_COMPRESSED,
+QCOW2_SUBCLUSTER_INVALID,
+} QCow2SubclusterType;
+
 typedef enum QCow2MetadataOverlap {
 

[PATCH v5 27/31] qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()

[Alberto Garcia]

This works now at the subcluster level and pwrite_zeroes_alignment is
updated accordingly.

qcow2_cluster_zeroize() is turned into qcow2_subcluster_zeroize() with
the following changes:

   - The request can now be subcluster-aligned.

   - The cluster-aligned body of the request is still zeroized using
 zero_in_l2_slice() as before.

   - The subcluster-aligned head and tail of the request are zeroized
 with the new zero_l2_subclusters() function.

There is just one thing to take into account for a possible future
improvement: compressed clusters cannot be partially zeroized so
zero_l2_subclusters() on the head or the tail can return -ENOTSUP.
This makes the caller repeat the *complete* request and write actual
zeroes to disk. This is sub-optimal because

   1) if the head area was compressed we would still be able to use
  the fast path for the body and possibly the tail.

   2) if the tail area was compressed we are writing zeroes to the
  head and the body areas, which are already zeroized.

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h |  4 +--
 block/qcow2-cluster.c | 80 +++
 block/qcow2.c | 27 ---
 3 files changed, 90 insertions(+), 21 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 05e3ef0ece..7349c6ce40 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -881,8 +881,8 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, 
QCowL2Meta *m);
 int qcow2_cluster_discard(BlockDriverState *bs, uint64_t offset,
   uint64_t bytes, enum qcow2_discard_type type,
   bool full_discard);
-int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,
-  uint64_t bytes, int flags);
+int qcow2_subcluster_zeroize(BlockDriverState *bs, uint64_t offset,
+ uint64_t bytes, int flags);
 
 int qcow2_expand_zero_clusters(BlockDriverState *bs,
BlockDriverAmendStatusCB *status_cb,
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 0a295076a3..d0cf9d52e6 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1971,12 +1971,58 @@ static int zero_in_l2_slice(BlockDriverState *bs, 
uint64_t offset,
 return nb_clusters;
 }
 
-int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,
-  uint64_t bytes, int flags)
+static int zero_l2_subclusters(BlockDriverState *bs, uint64_t offset,
+   unsigned nb_subclusters)
+{
+BDRVQcow2State *s = bs->opaque;
+uint64_t *l2_slice;
+uint64_t old_l2_bitmap, l2_bitmap;
+int l2_index, ret, sc = offset_to_sc_index(s, offset);
+
+/* For full clusters use zero_in_l2_slice() instead */
+assert(nb_subclusters > 0 && nb_subclusters < s->subclusters_per_cluster);
+assert(sc + nb_subclusters <= s->subclusters_per_cluster);
+
+ret = get_cluster_table(bs, offset, _slice, _index);
+if (ret < 0) {
+return ret;
+}
+
+switch (qcow2_get_cluster_type(bs, get_l2_entry(s, l2_slice, l2_index))) {
+case QCOW2_CLUSTER_COMPRESSED:
+ret = -ENOTSUP; /* We cannot partially zeroize compressed clusters */
+goto out;
+case QCOW2_CLUSTER_NORMAL:
+case QCOW2_CLUSTER_UNALLOCATED:
+break;
+default:
+g_assert_not_reached();
+}
+
+old_l2_bitmap = l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index);
+
+l2_bitmap |=  QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters - 1);
+l2_bitmap &= ~QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters - 1);
+
+if (old_l2_bitmap != l2_bitmap) {
+set_l2_bitmap(s, l2_slice, l2_index, l2_bitmap);
+qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice);
+}
+
+ret = 0;
+out:
+qcow2_cache_put(s->l2_table_cache, (void **) _slice);
+
+return ret;
+}
+
+int qcow2_subcluster_zeroize(BlockDriverState *bs, uint64_t offset,
+ uint64_t bytes, int flags)
 {
 BDRVQcow2State *s = bs->opaque;
 uint64_t end_offset = offset + bytes;
 uint64_t nb_clusters;
+unsigned head, tail;
 int64_t cleared;
 int ret;
 
@@ -1991,8 +2037,8 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t 
offset,
 }
 
 /* Caller must pass aligned values, except at image end */
-assert(QEMU_IS_ALIGNED(offset, s->cluster_size));
-assert(QEMU_IS_ALIGNED(end_offset, s->cluster_size) ||
+assert(offset_into_subcluster(s, offset) == 0);
+assert(offset_into_subcluster(s, end_offset) == 0 ||
end_offset >= bs->total_sectors << BDRV_SECTOR_BITS);
 
 /* The zero flag is only supported by version 3 and newer */
@@ -2000,11 +2046,26 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, 
uint64_t offset,
 return -ENOTSUP;
 }
 
-/* Each L2 slice is handled by its own loop iteration */
-nb_clusters = size_to_clusters(s, bytes);
+head = MIN(end_offset, ROUND_UP(offset, 

[PATCH v5 20/31] qcow2: Add subcluster support to qcow2_get_host_offset()

[Alberto Garcia]

The logic of this function remains pretty much the same, except that
it uses count_contiguous_subclusters(), which combines the logic of
count_contiguous_clusters() / count_contiguous_clusters_unallocated()
and checks individual subclusters.

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h |  38 +---
 block/qcow2-cluster.c | 141 --
 2 files changed, 82 insertions(+), 97 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 1663b5359c..05e3ef0ece 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -692,29 +692,6 @@ static inline QCow2ClusterType 
qcow2_get_cluster_type(BlockDriverState *bs,
 }
 }
 
-/*
- * For an image without extended L2 entries, return the
- * QCow2SubclusterType equivalent of a given QCow2ClusterType.
- */
-static inline
-QCow2SubclusterType qcow2_cluster_to_subcluster_type(QCow2ClusterType type)
-{
-switch (type) {
-case QCOW2_CLUSTER_COMPRESSED:
-return QCOW2_SUBCLUSTER_COMPRESSED;
-case QCOW2_CLUSTER_ZERO_PLAIN:
-return QCOW2_SUBCLUSTER_ZERO_PLAIN;
-case QCOW2_CLUSTER_ZERO_ALLOC:
-return QCOW2_SUBCLUSTER_ZERO_ALLOC;
-case QCOW2_CLUSTER_NORMAL:
-return QCOW2_SUBCLUSTER_NORMAL;
-case QCOW2_CLUSTER_UNALLOCATED:
-return QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN;
-default:
-g_assert_not_reached();
-}
-}
-
 /*
  * In an image without subsclusters @l2_bitmap is ignored and
  * @sc_index must be 0.
@@ -759,7 +736,20 @@ QCow2SubclusterType 
qcow2_get_subcluster_type(BlockDriverState *bs,
 g_assert_not_reached();
 }
 } else {
-return qcow2_cluster_to_subcluster_type(type);
+switch (type) {
+case QCOW2_CLUSTER_COMPRESSED:
+return QCOW2_SUBCLUSTER_COMPRESSED;
+case QCOW2_CLUSTER_ZERO_PLAIN:
+return QCOW2_SUBCLUSTER_ZERO_PLAIN;
+case QCOW2_CLUSTER_ZERO_ALLOC:
+return QCOW2_SUBCLUSTER_ZERO_ALLOC;
+case QCOW2_CLUSTER_NORMAL:
+return QCOW2_SUBCLUSTER_NORMAL;
+case QCOW2_CLUSTER_UNALLOCATED:
+return QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN;
+default:
+g_assert_not_reached();
+}
 }
 }
 
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index ffcb11edda..f500cbfb8e 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -376,66 +376,58 @@ fail:
 }
 
 /*
- * Checks how many clusters in a given L2 slice are contiguous in the image
- * file. As soon as one of the flags in the bitmask stop_flags changes compared
- * to the first cluster, the search is stopped and the cluster is not counted
- * as contiguous. (This allows it, for example, to stop at the first compressed
- * cluster which may require a different handling)
+ * Return the number of contiguous subclusters of the exact same type
+ * in a given L2 slice, starting from cluster @l2_index, subcluster
+ * @sc_index. Allocated subclusters are required to be contiguous in
+ * the image file.
+ * At most @nb_clusters are checked (note that this means clusters,
+ * not subclusters).
+ * Compressed clusters are always processed one by one but for the
+ * purpose of this count they are treated as if they were divided into
+ * subclusters of size s->subcluster_size.
  */
-static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters,
-int cluster_size, uint64_t *l2_slice, int l2_index, uint64_t 
stop_flags)
+static int count_contiguous_subclusters(BlockDriverState *bs, int nb_clusters,
+unsigned sc_index, uint64_t *l2_slice,
+int l2_index)
 {
 BDRVQcow2State *s = bs->opaque;
-int i;
-QCow2ClusterType first_cluster_type;
-uint64_t mask = stop_flags | L2E_OFFSET_MASK | QCOW_OFLAG_COMPRESSED;
-uint64_t first_entry = get_l2_entry(s, l2_slice, l2_index);
-uint64_t offset = first_entry & mask;
-
-first_cluster_type = qcow2_get_cluster_type(bs, first_entry);
-if (first_cluster_type == QCOW2_CLUSTER_UNALLOCATED) {
-return 0;
+int i, j, count = 0;
+uint64_t l2_entry = get_l2_entry(s, l2_slice, l2_index);
+uint64_t l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index);
+uint64_t expected_offset = l2_entry & L2E_OFFSET_MASK;
+bool check_offset = true;
+QCow2SubclusterType type =
+qcow2_get_subcluster_type(bs, l2_entry, l2_bitmap, sc_index);
+
+assert(type != QCOW2_SUBCLUSTER_INVALID); /* The caller should check this 
*/
+assert(l2_index + nb_clusters <= s->l2_size);
+
+if (type == QCOW2_SUBCLUSTER_COMPRESSED) {
+/* Compressed clusters are always processed one by one */
+return s->subclusters_per_cluster - sc_index;
 }
 
-/* must be allocated */
-assert(first_cluster_type == QCOW2_CLUSTER_NORMAL ||
-   first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC);
-
-for (i = 0; i < nb_clusters; i++) {
-uint64_t l2_entry = get_l2_entry(s, 

[PATCH v5 07/31] qcow2: Document the Extended L2 Entries feature

[Alberto Garcia]

Subcluster allocation in qcow2 is implemented by extending the
existing L2 table entries and adding additional information to
indicate the allocation status of each subcluster.

This patch documents the changes to the qcow2 format and how they
affect the calculation of the L2 cache size.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
 docs/interop/qcow2.txt | 68 --
 docs/qcow2-cache.txt   | 19 +++-
 2 files changed, 83 insertions(+), 4 deletions(-)

diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 298a031310..f08e43f228 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -42,6 +42,9 @@ The first cluster of a qcow2 image contains the file header:
 as the maximum cluster size and won't be able to open 
images
 with larger cluster sizes.
 
+Note: if the image has Extended L2 Entries then 
cluster_bits
+must be at least 14 (i.e. 16384 byte clusters).
+
  24 - 31:   size
 Virtual disk size in bytes.
 
@@ -117,7 +120,12 @@ the next fields through header_length.
 clusters. The compression_type field must be
 present and not zero.
 
-Bits 4-63:  Reserved (set to 0)
+Bit 4:  Extended L2 Entries.  If this bit is set then
+L2 table entries use an extended format that
+allows subcluster-based allocation. See the
+Extended L2 Entries section for more details.
+
+Bits 5-63:  Reserved (set to 0)
 
  80 -  87:  compatible_features
 Bitmask of compatible features. An implementation can
@@ -497,7 +505,7 @@ cannot be relaxed without an incompatible layout change).
 Given an offset into the virtual disk, the offset into the image file can be
 obtained as follows:
 
-l2_entries = (cluster_size / sizeof(uint64_t))
+l2_entries = (cluster_size / sizeof(uint64_t))[*]
 
 l2_index = (offset / cluster_size) % l2_entries
 l1_index = (offset / cluster_size) / l2_entries
@@ -507,6 +515,8 @@ obtained as follows:
 
 return cluster_offset + (offset % cluster_size)
 
+[*] this changes if Extended L2 Entries are enabled, see next section
+
 L1 table entry:
 
 Bit  0 -  8:Reserved (set to 0)
@@ -547,7 +557,8 @@ Standard Cluster Descriptor:
 nor is data read from the backing file if the cluster is
 unallocated.
 
-With version 2, this is always 0.
+With version 2 or with extended L2 entries (see the next
+section), this is always 0.
 
  1 -  8:Reserved (set to 0)
 
@@ -584,6 +595,57 @@ file (except if bit 0 in the Standard Cluster Descriptor 
is set). If there is
 no backing file or the backing file is smaller than the image, they shall read
 zeros for all parts that are not covered by the backing file.
 
+== Extended L2 Entries ==
+
+An image uses Extended L2 Entries if bit 4 is set on the incompatible_features
+field of the header.
+
+In these images standard data clusters are divided into 32 subclusters of the
+same size. They are contiguous and start from the beginning of the cluster.
+Subclusters can be allocated independently and the L2 entry contains 
information
+indicating the status of each one of them. Compressed data clusters don't have
+subclusters so they are treated the same as in images without this feature.
+
+The size of an extended L2 entry is 128 bits so the number of entries per table
+is calculated using this formula:
+
+l2_entries = (cluster_size / (2 * sizeof(uint64_t)))
+
+The first 64 bits have the same format as the standard L2 table entry described
+in the previous section, with the exception of bit 0 of the standard cluster
+descriptor.
+
+The last 64 bits contain a subcluster allocation bitmap with this format:
+
+Subcluster Allocation Bitmap (for standard clusters):
+
+Bit  0 -  31:   Allocation status (one bit per subcluster)
+
+1: the subcluster is allocated. In this case the
+   host cluster offset field must contain a valid
+   offset.
+0: the subcluster is not allocated. In this case
+   read requests shall go to the backing file or
+   return zeros if there is no backing file data.
+
+Bits are assigned starting from the least significant
+one (i.e. bit x is used for subcluster x).
+
+32 -  63Subcluster reads as zeros (one bit per subcluster)
+
+1: the subcluster reads as zeros. In this case the
+   allocation status bit must be unset. The host
+   cluster offset field may or 

[PATCH v5 12/31] qcow2: Add l2_entry_size()

[Alberto Garcia]

qcow2 images with subclusters have 128-bit L2 entries. The first 64
bits contain the same information as traditional images and the last
64 bits form a bitmap with the status of each individual subcluster.

Because of that we cannot assume that L2 entries are sizeof(uint64_t)
anymore. This function returns the proper value for the image.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
---
 block/qcow2.h  |  9 +
 block/qcow2-cluster.c  | 12 ++--
 block/qcow2-refcount.c | 14 --
 block/qcow2.c  |  8 
 4 files changed, 27 insertions(+), 16 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 8b1ed1cbcf..80ceb352c9 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -80,6 +80,10 @@
 
 #define QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER 32
 
+/* Size of normal and extended L2 entries */
+#define L2E_SIZE_NORMAL   (sizeof(uint64_t))
+#define L2E_SIZE_EXTENDED (sizeof(uint64_t) * 2)
+
 #define MIN_CLUSTER_BITS 9
 #define MAX_CLUSTER_BITS 21
 
@@ -503,6 +507,11 @@ static inline bool has_subclusters(BDRVQcow2State *s)
 return false;
 }
 
+static inline size_t l2_entry_size(BDRVQcow2State *s)
+{
+return has_subclusters(s) ? L2E_SIZE_EXTENDED : L2E_SIZE_NORMAL;
+}
+
 static inline uint64_t get_l2_entry(BDRVQcow2State *s, uint64_t *l2_slice,
 int idx)
 {
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 76fd0f3cdb..8b2fc550b7 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -208,7 +208,7 @@ static int l2_load(BlockDriverState *bs, uint64_t offset,
uint64_t l2_offset, uint64_t **l2_slice)
 {
 BDRVQcow2State *s = bs->opaque;
-int start_of_slice = sizeof(uint64_t) *
+int start_of_slice = l2_entry_size(s) *
 (offset_to_l2_index(s, offset) - offset_to_l2_slice_index(s, offset));
 
 return qcow2_cache_get(bs, s->l2_table_cache, l2_offset + start_of_slice,
@@ -281,7 +281,7 @@ static int l2_allocate(BlockDriverState *bs, int l1_index)
 
 /* allocate a new l2 entry */
 
-l2_offset = qcow2_alloc_clusters(bs, s->l2_size * sizeof(uint64_t));
+l2_offset = qcow2_alloc_clusters(bs, s->l2_size * l2_entry_size(s));
 if (l2_offset < 0) {
 ret = l2_offset;
 goto fail;
@@ -305,7 +305,7 @@ static int l2_allocate(BlockDriverState *bs, int l1_index)
 
 /* allocate a new entry in the l2 cache */
 
-slice_size2 = s->l2_slice_size * sizeof(uint64_t);
+slice_size2 = s->l2_slice_size * l2_entry_size(s);
 n_slices = s->cluster_size / slice_size2;
 
 trace_qcow2_l2_allocate_get_empty(bs, l1_index);
@@ -369,7 +369,7 @@ fail:
 }
 s->l1_table[l1_index] = old_l2_offset;
 if (l2_offset > 0) {
-qcow2_free_clusters(bs, l2_offset, s->l2_size * sizeof(uint64_t),
+qcow2_free_clusters(bs, l2_offset, s->l2_size * l2_entry_size(s),
 QCOW2_DISCARD_ALWAYS);
 }
 return ret;
@@ -716,7 +716,7 @@ static int get_cluster_table(BlockDriverState *bs, uint64_t 
offset,
 
 /* Then decrease the refcount of the old table */
 if (l2_offset) {
-qcow2_free_clusters(bs, l2_offset, s->l2_size * sizeof(uint64_t),
+qcow2_free_clusters(bs, l2_offset, s->l2_size * l2_entry_size(s),
 QCOW2_DISCARD_OTHER);
 }
 
@@ -1913,7 +1913,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState 
*bs, uint64_t *l1_table,
 int ret;
 int i, j;
 
-slice_size2 = s->l2_slice_size * sizeof(uint64_t);
+slice_size2 = s->l2_slice_size * l2_entry_size(s);
 n_slices = s->cluster_size / slice_size2;
 
 if (!is_active_l1) {
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index b299b9bda4..dfdcdd3c25 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1254,7 +1254,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
 l2_slice = NULL;
 l1_table = NULL;
 l1_size2 = l1_size * sizeof(uint64_t);
-slice_size2 = s->l2_slice_size * sizeof(uint64_t);
+slice_size2 = s->l2_slice_size * l2_entry_size(s);
 n_slices = s->cluster_size / slice_size2;
 
 s->cache_discards = true;
@@ -1605,7 +1605,7 @@ static int check_refcounts_l2(BlockDriverState *bs, 
BdrvCheckResult *res,
 int i, l2_size, nb_csectors, ret;
 
 /* Read L2 table from disk */
-l2_size = s->l2_size * sizeof(uint64_t);
+l2_size = s->l2_size * l2_entry_size(s);
 l2_table = g_malloc(l2_size);
 
 ret = bdrv_pread(bs->file, l2_offset, l2_table, l2_size);
@@ -1680,15 +1680,16 @@ static int check_refcounts_l2(BlockDriverState *bs, 
BdrvCheckResult *res,
 fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR",
 offset);
 if (fix & BDRV_FIX_ERRORS) {
+int idx = i * (l2_entry_size(s) / sizeof(uint64_t));
 uint64_t l2e_offset =
-l2_offset + 

[PATCH v5 13/31] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

[Alberto Garcia]

Extended L2 entries are 128-bit wide: 64 bits for the entry itself and
64 bits for the subcluster allocation bitmap.

In order to support them correctly get/set_l2_entry() need to be
updated so they take the entry width into account in order to
calculate the correct offset.

This patch also adds the get/set_l2_bitmap() functions that are
used to access the bitmaps. For convenience we allow calling
get_l2_bitmap() on images without subclusters. In this case the
returned value is always 0 and has no meaning.

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h | 21 +
 1 file changed, 21 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h
index 80ceb352c9..4ad93772b9 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -515,15 +515,36 @@ static inline size_t l2_entry_size(BDRVQcow2State *s)
 static inline uint64_t get_l2_entry(BDRVQcow2State *s, uint64_t *l2_slice,
 int idx)
 {
+idx *= l2_entry_size(s) / sizeof(uint64_t);
 return be64_to_cpu(l2_slice[idx]);
 }
 
+static inline uint64_t get_l2_bitmap(BDRVQcow2State *s, uint64_t *l2_slice,
+ int idx)
+{
+if (has_subclusters(s)) {
+idx *= l2_entry_size(s) / sizeof(uint64_t);
+return be64_to_cpu(l2_slice[idx + 1]);
+} else {
+return 0; /* For convenience only; this value has no meaning. */
+}
+}
+
 static inline void set_l2_entry(BDRVQcow2State *s, uint64_t *l2_slice,
 int idx, uint64_t entry)
 {
+idx *= l2_entry_size(s) / sizeof(uint64_t);
 l2_slice[idx] = cpu_to_be64(entry);
 }
 
+static inline void set_l2_bitmap(BDRVQcow2State *s, uint64_t *l2_slice,
+ int idx, uint64_t bitmap)
+{
+assert(has_subclusters(s));
+idx *= l2_entry_size(s) / sizeof(uint64_t);
+l2_slice[idx + 1] = cpu_to_be64(bitmap);
+}
+
 static inline bool has_data_file(BlockDriverState *bs)
 {
 BDRVQcow2State *s = bs->opaque;
-- 
2.20.1

[PATCH v5 09/31] qcow2: Add subcluster-related fields to BDRVQcow2State

[Alberto Garcia]

This patch adds the following new fields to BDRVQcow2State:

- subclusters_per_cluster: Number of subclusters in a cluster
- subcluster_size: The size of each subcluster, in bytes
- subcluster_bits: No. of bits so 1 << subcluster_bits = subcluster_size

Images without subclusters are treated as if they had exactly one
subcluster per cluster (i.e. subcluster_size = cluster_size).

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.h | 5 +
 block/qcow2.c | 5 +
 2 files changed, 10 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h
index 5e8036c3dd..9d8ca8068c 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -78,6 +78,8 @@
 /* The cluster reads as all zeros */
 #define QCOW_OFLAG_ZERO (1ULL << 0)
 
+#define QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER 32
+
 #define MIN_CLUSTER_BITS 9
 #define MAX_CLUSTER_BITS 21
 
@@ -284,6 +286,9 @@ typedef struct BDRVQcow2State {
 int cluster_bits;
 int cluster_size;
 int l2_slice_size;
+int subcluster_bits;
+int subcluster_size;
+int subclusters_per_cluster;
 int l2_bits;
 int l2_size;
 int l1_size;
diff --git a/block/qcow2.c b/block/qcow2.c
index 951fc19dd2..e5ae8aaff7 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1380,6 +1380,11 @@ static int coroutine_fn qcow2_do_open(BlockDriverState 
*bs, QDict *options,
 }
 }
 
+s->subclusters_per_cluster =
+has_subclusters(s) ? QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER : 1;
+s->subcluster_size = s->cluster_size / s->subclusters_per_cluster;
+s->subcluster_bits = ctz32(s->subcluster_size);
+
 /* Check support for various header values */
 if (header.refcount_order > 6) {
 error_setg(errp, "Reference count entry width too large; may not "
-- 
2.20.1

[PATCH v5 29/31] qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters

[Alberto Garcia]

This function is only used by qcow2_expand_zero_clusters() to
downgrade a qcow2 image to a previous version. It is however not
possible to downgrade an image with extended L2 entries because older
versions of qcow2 do not have this feature.

Signed-off-by: Alberto Garcia 
---
 block/qcow2-cluster.c  | 8 +++-
 tests/qemu-iotests/061 | 6 ++
 tests/qemu-iotests/061.out | 5 +
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index d0cf9d52e6..50da38800e 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -2113,6 +2113,9 @@ static int expand_zero_clusters_in_l1(BlockDriverState 
*bs, uint64_t *l1_table,
 int ret;
 int i, j;
 
+/* qcow2_downgrade() is not allowed in images with subclusters */
+assert(!has_subclusters(s));
+
 slice_size2 = s->l2_slice_size * l2_entry_size(s);
 n_slices = s->cluster_size / slice_size2;
 
@@ -2181,7 +2184,8 @@ static int expand_zero_clusters_in_l1(BlockDriverState 
*bs, uint64_t *l1_table,
 if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN) {
 if (!bs->backing) {
 /* not backed; therefore we can simply deallocate the
- * cluster */
+ * cluster. No need to call set_l2_bitmap(), this
+ * function doesn't support images with subclusters. */
 set_l2_entry(s, l2_slice, j, 0);
 l2_dirty = true;
 continue;
@@ -2252,6 +2256,8 @@ static int expand_zero_clusters_in_l1(BlockDriverState 
*bs, uint64_t *l1_table,
 } else {
 set_l2_entry(s, l2_slice, j, offset);
 }
+/* No need to call set_l2_bitmap() after set_l2_entry() because
+ * this function doesn't support images with subclusters. */
 l2_dirty = true;
 }
 
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index ce285d3084..f746d7ece3 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -268,6 +268,12 @@ $QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
 _img_info --format-specific
 _check_test_img
 
+echo
+echo "=== Testing version downgrade with extended L2 entries ==="
+echo
+_make_test_img -o "compat=1.1,extended_l2=on" 64M
+$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
+
 echo
 echo "=== Try changing the external data file ==="
 echo
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index b7b2533e0a..4c87bb1a3f 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -499,6 +499,11 @@ Format specific information:
 extended l2: false
 No errors were found on the image.
 
+=== Testing version downgrade with extended L2 entries ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: Cannot downgrade an image with incompatible features 0x10 set
+
 === Try changing the external data file ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-- 
2.20.1

[PATCH v5 06/31] qcow2: Add get_l2_entry() and set_l2_entry()

[Alberto Garcia]

The size of an L2 entry is 64 bits, but if we want to have subclusters
we need extended L2 entries. This means that we have to access L2
tables and slices differently depending on whether an image has
extended L2 entries or not.

This patch replaces all l2_slice[] accesses with calls to
get_l2_entry() and set_l2_entry().

Signed-off-by: Alberto Garcia 
Reviewed-by: Eric Blake 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.h  | 12 
 block/qcow2-cluster.c  | 63 ++
 block/qcow2-refcount.c | 17 ++--
 3 files changed, 54 insertions(+), 38 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 37e4f79e39..97fbaba574 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -492,6 +492,18 @@ typedef enum QCow2MetadataOverlap {
 
 #define INV_OFFSET (-1ULL)
 
+static inline uint64_t get_l2_entry(BDRVQcow2State *s, uint64_t *l2_slice,
+int idx)
+{
+return be64_to_cpu(l2_slice[idx]);
+}
+
+static inline void set_l2_entry(BDRVQcow2State *s, uint64_t *l2_slice,
+int idx, uint64_t entry)
+{
+l2_slice[idx] = cpu_to_be64(entry);
+}
+
 static inline bool has_data_file(BlockDriverState *bs)
 {
 BDRVQcow2State *s = bs->opaque;
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index fce0be7a08..76fd0f3cdb 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -383,12 +383,13 @@ fail:
  * cluster which may require a different handling)
  */
 static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters,
-int cluster_size, uint64_t *l2_slice, uint64_t stop_flags)
+int cluster_size, uint64_t *l2_slice, int l2_index, uint64_t 
stop_flags)
 {
+BDRVQcow2State *s = bs->opaque;
 int i;
 QCow2ClusterType first_cluster_type;
 uint64_t mask = stop_flags | L2E_OFFSET_MASK | QCOW_OFLAG_COMPRESSED;
-uint64_t first_entry = be64_to_cpu(l2_slice[0]);
+uint64_t first_entry = get_l2_entry(s, l2_slice, l2_index);
 uint64_t offset = first_entry & mask;
 
 first_cluster_type = qcow2_get_cluster_type(bs, first_entry);
@@ -401,7 +402,7 @@ static int count_contiguous_clusters(BlockDriverState *bs, 
int nb_clusters,
first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC);
 
 for (i = 0; i < nb_clusters; i++) {
-uint64_t l2_entry = be64_to_cpu(l2_slice[i]) & mask;
+uint64_t l2_entry = get_l2_entry(s, l2_slice, l2_index + i) & mask;
 if (offset + (uint64_t) i * cluster_size != l2_entry) {
 break;
 }
@@ -417,14 +418,16 @@ static int count_contiguous_clusters(BlockDriverState 
*bs, int nb_clusters,
 static int count_contiguous_clusters_unallocated(BlockDriverState *bs,
  int nb_clusters,
  uint64_t *l2_slice,
+ int l2_index,
  QCow2ClusterType wanted_type)
 {
+BDRVQcow2State *s = bs->opaque;
 int i;
 
 assert(wanted_type == QCOW2_CLUSTER_ZERO_PLAIN ||
wanted_type == QCOW2_CLUSTER_UNALLOCATED);
 for (i = 0; i < nb_clusters; i++) {
-uint64_t entry = be64_to_cpu(l2_slice[i]);
+uint64_t entry = get_l2_entry(s, l2_slice, l2_index + i);
 QCow2ClusterType type = qcow2_get_cluster_type(bs, entry);
 
 if (type != wanted_type) {
@@ -573,7 +576,7 @@ int qcow2_get_host_offset(BlockDriverState *bs, uint64_t 
offset,
 /* find the cluster offset for the given disk offset */
 
 l2_index = offset_to_l2_slice_index(s, offset);
-l2_entry = be64_to_cpu(l2_slice[l2_index]);
+l2_entry = get_l2_entry(s, l2_slice, l2_index);
 
 nb_clusters = size_to_clusters(s, bytes_needed);
 /* bytes_needed <= *bytes + offset_in_cluster, both of which are unsigned
@@ -608,7 +611,7 @@ int qcow2_get_host_offset(BlockDriverState *bs, uint64_t 
offset,
 case QCOW2_CLUSTER_UNALLOCATED:
 /* how many empty clusters ? */
 c = count_contiguous_clusters_unallocated(bs, nb_clusters,
-  _slice[l2_index], type);
+  l2_slice, l2_index, type);
 *host_offset = 0;
 break;
 case QCOW2_CLUSTER_ZERO_ALLOC:
@@ -617,7 +620,7 @@ int qcow2_get_host_offset(BlockDriverState *bs, uint64_t 
offset,
 *host_offset = host_cluster_offset + offset_in_cluster;
 /* how many allocated clusters ? */
 c = count_contiguous_clusters(bs, nb_clusters, s->cluster_size,
-  _slice[l2_index], QCOW_OFLAG_ZERO);
+  l2_slice, l2_index, QCOW_OFLAG_ZERO);
 if (offset_into_cluster(s, host_cluster_offset)) {
 qcow2_signal_corruption(bs, true, -1, -1,
 "Cluster allocation offset %#"

[PATCH v5 04/31] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

[Alberto Garcia]

We are going to need it in other places.

Signed-off-by: Alberto Garcia 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
---
 block/qcow2-cluster.c | 34 +++---
 1 file changed, 19 insertions(+), 15 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 61ad638bdc..80f9787461 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1087,6 +1087,24 @@ static void calculate_l2_meta(BlockDriverState *bs,
 QLIST_INSERT_HEAD(>cluster_allocs, *m, next_in_flight);
 }
 
+/* Returns true if writing to a cluster requires COW */
+static bool cluster_needs_cow(BlockDriverState *bs, uint64_t l2_entry)
+{
+switch (qcow2_get_cluster_type(bs, l2_entry)) {
+case QCOW2_CLUSTER_NORMAL:
+if (l2_entry & QCOW_OFLAG_COPIED) {
+return false;
+}
+case QCOW2_CLUSTER_UNALLOCATED:
+case QCOW2_CLUSTER_COMPRESSED:
+case QCOW2_CLUSTER_ZERO_PLAIN:
+case QCOW2_CLUSTER_ZERO_ALLOC:
+return true;
+default:
+abort();
+}
+}
+
 /*
  * Returns the number of contiguous clusters that can be used for an allocating
  * write, but require COW to be performed (this includes yet unallocated space,
@@ -1099,25 +1117,11 @@ static int count_cow_clusters(BlockDriverState *bs, int 
nb_clusters,
 
 for (i = 0; i < nb_clusters; i++) {
 uint64_t l2_entry = be64_to_cpu(l2_slice[l2_index + i]);
-QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, l2_entry);
-
-switch(cluster_type) {
-case QCOW2_CLUSTER_NORMAL:
-if (l2_entry & QCOW_OFLAG_COPIED) {
-goto out;
-}
+if (!cluster_needs_cow(bs, l2_entry)) {
 break;
-case QCOW2_CLUSTER_UNALLOCATED:
-case QCOW2_CLUSTER_COMPRESSED:
-case QCOW2_CLUSTER_ZERO_PLAIN:
-case QCOW2_CLUSTER_ZERO_ALLOC:
-break;
-default:
-abort();
 }
 }
 
-out:
 assert(i <= nb_clusters);
 return i;
 }
-- 
2.20.1

[PATCH v5 22/31] qcow2: Add subcluster support to discard_in_l2_slice()

[Alberto Garcia]

Two things need to be taken into account here:

1) With full_discard == true the L2 entry must be cleared completely.
   This also includes the L2 bitmap if the image has extended L2
   entries.

2) With full_discard == false we have to make the discarded cluster
   read back as zeroes. With normal L2 entries this is done with the
   QCOW_OFLAG_ZERO bit, whereas with extended L2 entries this is done
   with the individual 'all zeroes' bits for each subcluster.

   Note however that QCOW_OFLAG_ZERO is not supported in v2 qcow2
   images so, if there is a backing file, discard cannot guarantee
   that the image will read back as zeroes. If this is important for
   the caller it should forbid it as qcow2_co_pdiscard() does (see
   80f5c01183 for more details).

Signed-off-by: Alberto Garcia 
---
 block/qcow2-cluster.c | 51 +++
 1 file changed, 22 insertions(+), 29 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index bf32ed0825..2283a308d0 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1804,11 +1804,16 @@ static int discard_in_l2_slice(BlockDriverState *bs, 
uint64_t offset,
 assert(nb_clusters <= INT_MAX);
 
 for (i = 0; i < nb_clusters; i++) {
-uint64_t old_l2_entry;
-
-old_l2_entry = get_l2_entry(s, l2_slice, l2_index + i);
+uint64_t old_l2_entry = get_l2_entry(s, l2_slice, l2_index + i);
+uint64_t old_l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index + i);
+uint64_t new_l2_entry = old_l2_entry;
+uint64_t new_l2_bitmap = old_l2_bitmap;
+QCow2ClusterType type = qcow2_get_cluster_type(bs, old_l2_entry);
 
 /*
+ * If full_discard is true, the cluster should not read back as zeroes,
+ * but rather fall through to the backing file.
+ *
  * If full_discard is false, make sure that a discarded area reads back
  * as zeroes for v3 images (we cannot do it for v2 without actually
  * writing a zero-filled buffer). We can skip the operation if the
@@ -1817,40 +1822,28 @@ static int discard_in_l2_slice(BlockDriverState *bs, 
uint64_t offset,
  *
  * TODO We might want to use bdrv_block_status(bs) here, but we're
  * holding s->lock, so that doesn't work today.
- *
- * If full_discard is true, the sector should not read back as zeroes,
- * but rather fall through to the backing file.
  */
-switch (qcow2_get_cluster_type(bs, old_l2_entry)) {
-case QCOW2_CLUSTER_UNALLOCATED:
-if (full_discard || !bs->backing) {
-continue;
+if (full_discard) {
+new_l2_entry = new_l2_bitmap = 0;
+} else if (bs->backing || qcow2_cluster_is_allocated(type)) {
+if (has_subclusters(s)) {
+new_l2_entry = 0;
+new_l2_bitmap = QCOW_L2_BITMAP_ALL_ZEROES;
+} else {
+new_l2_entry = s->qcow_version >= 3 ? QCOW_OFLAG_ZERO : 0;
 }
-break;
+}
 
-case QCOW2_CLUSTER_ZERO_PLAIN:
-if (!full_discard) {
-continue;
-}
-break;
-
-case QCOW2_CLUSTER_ZERO_ALLOC:
-case QCOW2_CLUSTER_NORMAL:
-case QCOW2_CLUSTER_COMPRESSED:
-break;
-
-default:
-abort();
+if (old_l2_entry == new_l2_entry && old_l2_bitmap == new_l2_bitmap) {
+continue;
 }
 
 /* First remove L2 entries */
 qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice);
-if (!full_discard && s->qcow_version >= 3) {
-set_l2_entry(s, l2_slice, l2_index + i, QCOW_OFLAG_ZERO);
-} else {
-set_l2_entry(s, l2_slice, l2_index + i, 0);
+set_l2_entry(s, l2_slice, l2_index + i, new_l2_entry);
+if (has_subclusters(s)) {
+set_l2_bitmap(s, l2_slice, l2_index + i, new_l2_bitmap);
 }
-
 /* Then decrease the refcount */
 qcow2_free_any_clusters(bs, old_l2_entry, 1, type);
 }
-- 
2.20.1

[PATCH v5 15/31] qcow2: Add qcow2_cluster_is_allocated()

[Alberto Garcia]

This helper function tells us if a cluster is allocated (that is,
there is an associated host offset for it).

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h
index be7816a3b8..b5db8d2f36 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -763,6 +763,12 @@ QCow2SubclusterType 
qcow2_get_subcluster_type(BlockDriverState *bs,
 }
 }
 
+static inline bool qcow2_cluster_is_allocated(QCow2ClusterType type)
+{
+return (type == QCOW2_CLUSTER_COMPRESSED || type == QCOW2_CLUSTER_NORMAL ||
+type == QCOW2_CLUSTER_ZERO_ALLOC);
+}
+
 /* Check whether refcounts are eager or lazy */
 static inline bool qcow2_need_accurate_refcounts(BDRVQcow2State *s)
 {
-- 
2.20.1

[PATCH v5 02/31] qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()

[Alberto Garcia]

qcow2_get_cluster_offset() takes an (unaligned) guest offset and
returns the (aligned) offset of the corresponding cluster in the qcow2
image.

In practice none of the callers need to know where the cluster starts
so this patch makes the function calculate and return the final host
offset directly. The function is also renamed accordingly.

There is a pre-existing exception with compressed clusters: in this
case the function returns the complete cluster descriptor (containing
the offset and size of the compressed data). This does not change with
this patch but it is now documented.

Signed-off-by: Alberto Garcia 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.h |  4 ++--
 block/qcow2-cluster.c | 42 +++---
 block/qcow2.c | 24 +++-
 3 files changed, 32 insertions(+), 38 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index f4de0a27d5..37e4f79e39 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -676,8 +676,8 @@ int qcow2_write_l1_entry(BlockDriverState *bs, int 
l1_index);
 int qcow2_encrypt_sectors(BDRVQcow2State *s, int64_t sector_num,
   uint8_t *buf, int nb_sectors, bool enc, Error 
**errp);
 
-int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
- unsigned int *bytes, uint64_t *cluster_offset);
+int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
+  unsigned int *bytes, uint64_t *host_offset);
 int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
unsigned int *bytes, uint64_t *host_offset,
QCowL2Meta **m);
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 4b5fc8c4a7..9ab41cb728 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -496,10 +496,15 @@ static int coroutine_fn 
do_perform_cow_write(BlockDriverState *bs,
 
 
 /*
- * get_cluster_offset
+ * get_host_offset
  *
- * For a given offset of the virtual disk, find the cluster type and offset in
- * the qcow2 file. The offset is stored in *cluster_offset.
+ * For a given offset of the virtual disk find the equivalent host
+ * offset in the qcow2 file and store it in *host_offset. Neither
+ * offset needs to be aligned to a cluster boundary.
+ *
+ * If the cluster is unallocated then *host_offset will be 0.
+ * If the cluster is compressed then *host_offset will contain the
+ * complete compressed cluster descriptor.
  *
  * On entry, *bytes is the maximum number of contiguous bytes starting at
  * offset that we are interested in.
@@ -511,12 +516,12 @@ static int coroutine_fn 
do_perform_cow_write(BlockDriverState *bs,
  * Returns the cluster type (QCOW2_CLUSTER_*) on success, -errno in error
  * cases.
  */
-int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
- unsigned int *bytes, uint64_t *cluster_offset)
+int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset,
+  unsigned int *bytes, uint64_t *host_offset)
 {
 BDRVQcow2State *s = bs->opaque;
 unsigned int l2_index;
-uint64_t l1_index, l2_offset, *l2_slice;
+uint64_t l1_index, l2_offset, *l2_slice, l2_entry;
 int c;
 unsigned int offset_in_cluster;
 uint64_t bytes_available, bytes_needed, nb_clusters;
@@ -537,8 +542,6 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t 
offset,
 bytes_needed = bytes_available;
 }
 
-*cluster_offset = 0;
-
 /* seek to the l2 offset in the l1 table */
 
 l1_index = offset_to_l1_index(s, offset);
@@ -570,7 +573,7 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t 
offset,
 /* find the cluster offset for the given disk offset */
 
 l2_index = offset_to_l2_slice_index(s, offset);
-*cluster_offset = be64_to_cpu(l2_slice[l2_index]);
+l2_entry = be64_to_cpu(l2_slice[l2_index]);
 
 nb_clusters = size_to_clusters(s, bytes_needed);
 /* bytes_needed <= *bytes + offset_in_cluster, both of which are unsigned
@@ -578,7 +581,7 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t 
offset,
  * true */
 assert(nb_clusters <= INT_MAX);
 
-type = qcow2_get_cluster_type(bs, *cluster_offset);
+type = qcow2_get_cluster_type(bs, l2_entry);
 if (s->qcow_version < 3 && (type == QCOW2_CLUSTER_ZERO_PLAIN ||
 type == QCOW2_CLUSTER_ZERO_ALLOC)) {
 qcow2_signal_corruption(bs, true, -1, -1, "Zero cluster entry found"
@@ -599,42 +602,43 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, 
uint64_t offset,
 }
 /* Compressed clusters can only be processed one by one */
 c = 1;
-*cluster_offset &= L2E_COMPRESSED_OFFSET_SIZE_MASK;
+*host_offset = l2_entry & L2E_COMPRESSED_OFFSET_SIZE_MASK;
 break;
 case QCOW2_CLUSTER_ZERO_PLAIN:
 case QCOW2_CLUSTER_UNALLOCATED:
 /* how many empty clusters ? */
 

[PATCH v5 21/31] qcow2: Add subcluster support to zero_in_l2_slice()

[Alberto Garcia]

The QCOW_OFLAG_ZERO bit that indicates that a cluster reads as
zeroes is only used in standard L2 entries. Extended L2 entries use
individual 'all zeroes' bits for each subcluster.

This must be taken into account when updating the L2 entry and also
when deciding that an existing entry does not need to be updated.

Signed-off-by: Alberto Garcia 
---
 block/qcow2-cluster.c | 36 +++-
 1 file changed, 19 insertions(+), 17 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index f500cbfb8e..bf32ed0825 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1913,7 +1913,6 @@ static int zero_in_l2_slice(BlockDriverState *bs, 
uint64_t offset,
 int l2_index;
 int ret;
 int i;
-bool unmap = !!(flags & BDRV_REQ_MAY_UNMAP);
 
 ret = get_cluster_table(bs, offset, _slice, _index);
 if (ret < 0) {
@@ -1925,28 +1924,31 @@ static int zero_in_l2_slice(BlockDriverState *bs, 
uint64_t offset,
 assert(nb_clusters <= INT_MAX);
 
 for (i = 0; i < nb_clusters; i++) {
-uint64_t old_offset;
-QCow2ClusterType cluster_type;
+uint64_t old_l2_entry = get_l2_entry(s, l2_slice, l2_index + i);
+uint64_t old_l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index + i);
+QCow2ClusterType type = qcow2_get_cluster_type(bs, old_l2_entry);
+bool unmap = (type == QCOW2_CLUSTER_COMPRESSED) ||
+((flags & BDRV_REQ_MAY_UNMAP) && qcow2_cluster_is_allocated(type));
+uint64_t new_l2_entry = unmap ? 0 : old_l2_entry;
+uint64_t new_l2_bitmap = old_l2_bitmap;
 
-old_offset = get_l2_entry(s, l2_slice, l2_index + i);
+if (has_subclusters(s)) {
+new_l2_bitmap = QCOW_L2_BITMAP_ALL_ZEROES;
+} else {
+new_l2_entry |= QCOW_OFLAG_ZERO;
+}
 
-/*
- * Minimize L2 changes if the cluster already reads back as
- * zeroes with correct allocation.
- */
-cluster_type = qcow2_get_cluster_type(bs, old_offset);
-if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN ||
-(cluster_type == QCOW2_CLUSTER_ZERO_ALLOC && !unmap)) {
+if (old_l2_entry == new_l2_entry && old_l2_bitmap == new_l2_bitmap) {
 continue;
 }
 
 qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice);
-if (cluster_type == QCOW2_CLUSTER_COMPRESSED || unmap) {
-set_l2_entry(s, l2_slice, l2_index + i, QCOW_OFLAG_ZERO);
-qcow2_free_any_clusters(bs, old_offset, 1, QCOW2_DISCARD_REQUEST);
-} else {
-uint64_t entry = get_l2_entry(s, l2_slice, l2_index + i);
-set_l2_entry(s, l2_slice, l2_index + i, entry | QCOW_OFLAG_ZERO);
+if (unmap) {
+qcow2_free_any_clusters(bs, old_l2_entry, 1, 
QCOW2_DISCARD_REQUEST);
+}
+set_l2_entry(s, l2_slice, l2_index + i, new_l2_entry);
+if (has_subclusters(s)) {
+set_l2_bitmap(s, l2_slice, l2_index + i, new_l2_bitmap);
 }
 }
 
-- 
2.20.1

[PATCH v5 01/31] qcow2: Make Qcow2AioTask store the full host offset

[Alberto Garcia]

The file_cluster_offset field of Qcow2AioTask stores a cluster-aligned
host offset. In practice this is not very useful because all users(*)
of this structure need the final host offset into the cluster, which
they calculate using

   host_offset = file_cluster_offset + offset_into_cluster(s, offset)

There is no reason why Qcow2AioTask cannot store host_offset directly
and that is what this patch does.

(*) compressed clusters are the exception: in this case what
file_cluster_offset was storing was the full compressed cluster
descriptor (offset + size). This does not change with this patch
but it is documented now.

Signed-off-by: Alberto Garcia 
Reviewed-by: Eric Blake 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.c  | 69 ++
 block/trace-events |  2 +-
 2 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 8c97b06783..a387809aa9 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -74,7 +74,7 @@ typedef struct {
 
 static int coroutine_fn
 qcow2_co_preadv_compressed(BlockDriverState *bs,
-   uint64_t file_cluster_offset,
+   uint64_t cluster_descriptor,
uint64_t offset,
uint64_t bytes,
QEMUIOVector *qiov,
@@ -2043,7 +2043,7 @@ out:
 
 static coroutine_fn int
 qcow2_co_preadv_encrypted(BlockDriverState *bs,
-   uint64_t file_cluster_offset,
+   uint64_t host_offset,
uint64_t offset,
uint64_t bytes,
QEMUIOVector *qiov,
@@ -2070,16 +2070,12 @@ qcow2_co_preadv_encrypted(BlockDriverState *bs,
 }
 
 BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO);
-ret = bdrv_co_pread(s->data_file,
-file_cluster_offset + offset_into_cluster(s, offset),
-bytes, buf, 0);
+ret = bdrv_co_pread(s->data_file, host_offset, bytes, buf, 0);
 if (ret < 0) {
 goto fail;
 }
 
-if (qcow2_co_decrypt(bs,
- file_cluster_offset + offset_into_cluster(s, offset),
- offset, buf, bytes) < 0)
+if (qcow2_co_decrypt(bs, host_offset, offset, buf, bytes) < 0)
 {
 ret = -EIO;
 goto fail;
@@ -2097,7 +2093,7 @@ typedef struct Qcow2AioTask {
 
 BlockDriverState *bs;
 QCow2ClusterType cluster_type; /* only for read */
-uint64_t file_cluster_offset;
+uint64_t host_offset; /* or full descriptor in compressed clusters */
 uint64_t offset;
 uint64_t bytes;
 QEMUIOVector *qiov;
@@ -2110,7 +2106,7 @@ static coroutine_fn int qcow2_add_task(BlockDriverState 
*bs,
AioTaskPool *pool,
AioTaskFunc func,
QCow2ClusterType cluster_type,
-   uint64_t file_cluster_offset,
+   uint64_t host_offset,
uint64_t offset,
uint64_t bytes,
QEMUIOVector *qiov,
@@ -2125,7 +2121,7 @@ static coroutine_fn int qcow2_add_task(BlockDriverState 
*bs,
 .bs = bs,
 .cluster_type = cluster_type,
 .qiov = qiov,
-.file_cluster_offset = file_cluster_offset,
+.host_offset = host_offset,
 .offset = offset,
 .bytes = bytes,
 .qiov_offset = qiov_offset,
@@ -2134,7 +2130,7 @@ static coroutine_fn int qcow2_add_task(BlockDriverState 
*bs,
 
 trace_qcow2_add_task(qemu_coroutine_self(), bs, pool,
  func == qcow2_co_preadv_task_entry ? "read" : "write",
- cluster_type, file_cluster_offset, offset, bytes,
+ cluster_type, host_offset, offset, bytes,
  qiov, qiov_offset);
 
 if (!pool) {
@@ -2148,13 +2144,12 @@ static coroutine_fn int qcow2_add_task(BlockDriverState 
*bs,
 
 static coroutine_fn int qcow2_co_preadv_task(BlockDriverState *bs,
  QCow2ClusterType cluster_type,
- uint64_t file_cluster_offset,
+ uint64_t host_offset,
  uint64_t offset, uint64_t bytes,
  QEMUIOVector *qiov,
  size_t qiov_offset)
 {
 BDRVQcow2State *s = bs->opaque;
-int offset_in_cluster = offset_into_cluster(s, offset);
 
 switch (cluster_type) {
 case QCOW2_CLUSTER_ZERO_PLAIN:
@@ -2170,19 +2165,17 @@ static coroutine_fn int 
qcow2_co_preadv_task(BlockDriverState *bs,
qiov, 

[PATCH v5 18/31] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

[Alberto Garcia]

When dealing with subcluster types there is a new value called
QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in
QCow2ClusterType.

This patch handles that value in all places where subcluster types
are processed.

Signed-off-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.c | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 13965f2e1d..63e952b89a 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1994,7 +1994,8 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 *pnum = bytes;
 
 if ((type == QCOW2_SUBCLUSTER_NORMAL ||
- type == QCOW2_SUBCLUSTER_ZERO_ALLOC) && !s->crypto) {
+ type == QCOW2_SUBCLUSTER_ZERO_ALLOC ||
+ type == QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC) && !s->crypto) {
 *map = host_offset;
 *file = s->data_file->bs;
 status |= BDRV_BLOCK_OFFSET_VALID;
@@ -2002,7 +2003,8 @@ static int coroutine_fn 
qcow2_co_block_status(BlockDriverState *bs,
 if (type == QCOW2_SUBCLUSTER_ZERO_PLAIN ||
 type == QCOW2_SUBCLUSTER_ZERO_ALLOC) {
 status |= BDRV_BLOCK_ZERO;
-} else if (type != QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN) {
+} else if (type != QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN &&
+   type != QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC) {
 status |= BDRV_BLOCK_DATA;
 }
 if (s->metadata_preallocation && (status & BDRV_BLOCK_DATA) &&
@@ -2165,6 +2167,7 @@ static coroutine_fn int 
qcow2_co_preadv_task(BlockDriverState *bs,
 g_assert_not_reached();
 
 case QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN:
+case QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC:
 assert(bs->backing); /* otherwise handled in qcow2_co_preadv_part */
 
 BLKDBG_EVENT(bs->file, BLKDBG_READ_BACKING_AIO);
@@ -2233,7 +2236,8 @@ static coroutine_fn int 
qcow2_co_preadv_part(BlockDriverState *bs,
 
 if (type == QCOW2_SUBCLUSTER_ZERO_PLAIN ||
 type == QCOW2_SUBCLUSTER_ZERO_ALLOC ||
-(type == QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN && !bs->backing))
+(type == QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN && !bs->backing) ||
+(type == QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC && !bs->backing))
 {
 qemu_iovec_memset(qiov, qiov_offset, 0, cur_bytes);
 } else {
@@ -3761,6 +3765,7 @@ static coroutine_fn int 
qcow2_co_pwrite_zeroes(BlockDriverState *bs,
 ret = qcow2_get_host_offset(bs, offset, , , );
 if (ret < 0 ||
 (type != QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN &&
+ type != QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC &&
  type != QCOW2_SUBCLUSTER_ZERO_PLAIN &&
  type != QCOW2_SUBCLUSTER_ZERO_ALLOC)) {
 qemu_co_mutex_unlock(>lock);
@@ -3839,6 +3844,7 @@ qcow2_co_copy_range_from(BlockDriverState *bs,
 
 switch (type) {
 case QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN:
+case QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC:
 if (bs->backing && bs->backing->bs) {
 int64_t backing_length = bdrv_getlength(bs->backing->bs);
 if (src_offset >= backing_length) {
-- 
2.20.1

[PATCH v5 11/31] qcow2: Add offset_into_subcluster() and size_to_subclusters()

[Alberto Garcia]

Like offset_into_cluster() and size_to_clusters(), but for
subclusters.

Signed-off-by: Alberto Garcia 
---
 block/qcow2.h | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h
index e68febb15b..8b1ed1cbcf 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -537,11 +537,21 @@ static inline int64_t offset_into_cluster(BDRVQcow2State 
*s, int64_t offset)
 return offset & (s->cluster_size - 1);
 }
 
+static inline int64_t offset_into_subcluster(BDRVQcow2State *s, int64_t offset)
+{
+return offset & (s->subcluster_size - 1);
+}
+
 static inline uint64_t size_to_clusters(BDRVQcow2State *s, uint64_t size)
 {
 return (size + (s->cluster_size - 1)) >> s->cluster_bits;
 }
 
+static inline uint64_t size_to_subclusters(BDRVQcow2State *s, uint64_t size)
+{
+return (size + (s->subcluster_size - 1)) >> s->subcluster_bits;
+}
+
 static inline int64_t size_to_l1(BDRVQcow2State *s, int64_t size)
 {
 int shift = s->cluster_bits + s->l2_bits;
-- 
2.20.1