Re: [Qemu-devel] [PATCH v3 00/16] qapi: Schema language cleanups & doc improvements

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190913201349.24332-1-arm...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v3 00/16] qapi: Schema language cleanups & doc 
improvements
Message-id: 20190913201349.24332-1-arm...@redhat.com
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
9491787 qapi: Tweak code to match docs/devel/qapi-code-gen.txt
6fb4a27 docs/devel/qapi-code-gen: Improve QAPI schema language doc
a474494 docs/devel/qapi-code-gen: Rewrite introduction to schema
6640272 docs/devel/qapi-code-gen: Rewrite compatibility considerations
d3cec78 docs/devel/qapi-code-gen: Reorder sections for readability
b48aba3 qapi: Adjust frontend errors to say enum value, not member
70ce619 qapi: Permit omitting all flat union branches
4ecd688 qapi: Permit alternates with just one branch
6fb8f9e qapi: Permit 'boxed' with empty type
c5e9097 qapi: Drop support for escape sequences other than \\
113836f qapi: Restrict strings to printable ASCII
fe760f3 tests/qapi-schema: Demonstrate bad reporting of funny characters
776df7c docs/devel/qapi-code-gen: Minor specification fixes
181a438 qapi: Drop support for boxed alternate arguments
57f9740 qapi: Drop check_type()'s redundant parameter @allow_optional
7975c5d scripts/git.orderfile: Match QAPI schema more precisely

=== OUTPUT BEGIN ===
1/16 Checking commit 7975c5d69cca (scripts/git.orderfile: Match QAPI schema 
more precisely)
2/16 Checking commit 57f9740b5d6c (qapi: Drop check_type()'s redundant 
parameter @allow_optional)
3/16 Checking commit 181a4384e8b8 (qapi: Drop support for boxed alternate 
arguments)
4/16 Checking commit 776df7cd45c6 (docs/devel/qapi-code-gen: Minor 
specification fixes)
5/16 Checking commit fe760f39461c (tests/qapi-schema: Demonstrate bad reporting 
of funny characters)
6/16 Checking commit 113836f31dd5 (qapi: Restrict strings to printable ASCII)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#109: 
new file mode 100644

ERROR: Invalid UTF-8, patch and commit message should be encoded in UTF-8
#126: FILE: tests/qapi-schema/string-code-point-127.json:2:
+{ 'command': '⌦' }
   ^

ERROR: Invalid UTF-8, patch and commit message should be encoded in UTF-8
#152: FILE: tests/qapi-schema/string-code-point-31.json:2:
+{ 'command': '␟' }
   ^

total: 2 errors, 1 warnings, 76 lines checked

Patch 6/16 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

7/16 Checking commit c5e90975c8c5 (qapi: Drop support for escape sequences 
other than \\)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#74: 
deleted file mode 100644

total: 0 errors, 1 warnings, 53 lines checked

Patch 7/16 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
8/16 Checking commit 6fb8f9e7baa7 (qapi: Permit 'boxed' with empty type)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#130: 
deleted file mode 100644

total: 0 errors, 1 warnings, 129 lines checked

Patch 8/16 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
9/16 Checking commit 4ecd688cd6a1 (qapi: Permit alternates with just one branch)
10/16 Checking commit 70ce619cf6af (qapi: Permit omitting all flat union 
branches)
11/16 Checking commit b48aba3a338c (qapi: Adjust frontend errors to say enum 
value, not member)
12/16 Checking commit d3cec7869722 (docs/devel/qapi-code-gen: Reorder sections 
for readability)
13/16 Checking commit 66402729d139 (docs/devel/qapi-code-gen: Rewrite 
compatibility considerations)
14/16 Checking commit a474494a010b (docs/devel/qapi-code-gen: Rewrite 
introduction to schema)
15/16 Checking commit 6fb4a273e1f6 (docs/devel/qapi-code-gen: Improve QAPI 
schema language doc)
16/16 Checking commit 9491787d3689 (qapi: Tweak code to match 
docs/devel/qapi-code-gen.txt)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190913201349.24332-1-arm...@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

[Qemu-devel] [Bug 1606899] Re: virtio-vga does not let guest poweroff properly

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1606899

Title:
  virtio-vga does not let guest poweroff properly

Status in QEMU:
  Expired

Bug description:
  I have a VM running rawhide (Fedora development) and I can't poweroff
  the machine when I enable virtio-vga. Reboot works correctly. Using
  QXL works also. The machine arrive to print the "Powering off" message
  (from Linux kernel) but then hangs.

  The command line is

  /usr/bin/qemu-system-x86_64 -machine accel=kvm -name rawhide -machine
  pc-i440fx-2.3,accel=kvm,usb=off -cpu Haswell-noTSX -m 2048 -realtime
  mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
  64216421-aec4-4ce4-aa52-aed9e4e31a1c -no-user-config -nodefaults
  -chardev
  socket,id=charmonitor,path=/var/lib/libvirt/qemu/rawhide.monitor,server,nowait
  -mon chardev=charmonitor,id=monitor,mode=control -rtc
  base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-
  hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
  PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-
  ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-
  uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
  -device ich9-usb-
  uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9
  -usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device
  virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
  file=/home/rawhide.qcow2,if=none,id=drive-virtio-disk0,format=qcow2
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-
  disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-
  ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-
  ide0-0-0,id=ide0-0-0 -netdev user,id=hostnet0 -device virtio-net-
  pci,netdev=hostnet0,id=net0,mac=52:54:00:fc:11:43,bus=pci.0,addr=0x3
  -chardev pty,id=charserial0 -device isa-
  serial,chardev=charserial0,id=serial0 -chardev
  
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rawhide.org.qemu.guest_agent.0,server,nowait
  -device virtserialport,bus=virtio-
  serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
  -chardev spicevmc,id=charchannel1,name=vdagent -device
  virtserialport,bus=virtio-
  serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0
  -device usb-tablet,id=input0 -spice ipv4,addr=0.0.0.0,port=5900
  ,disable-ticketing,image-compression=lz,seamless-migration=on
  ,streaming-video=filter -device virtio-vga,bus=pci.0,addr=0x2 -device
  intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-
  duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
  spicevmc,id=charredir0,name=usbredir -device usb-
  redir,chardev=charredir0,id=redir0 -chardev
  spicevmc,id=charredir1,name=usbredir -device usb-
  redir,chardev=charredir1,id=redir1 -device virtio-balloon-
  pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

  I though was due to Virgl but disabling it does not change.

  I'm using Qemu 2.6.0 from Fedora 24.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1606899/+subscriptions

[Qemu-devel] [Bug 1606708] Re: QEMU crashes when switching consoles using SDL

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1606708

Title:
  QEMU crashes when switching consoles using SDL

Status in QEMU:
  Expired

Bug description:
  I've trying to use QEMU with SDL, and I noticed that it doesn't behave
  well, specially when switching from VGA to any console. Resuming,
  switching is erratic when using SDL, and its effects go from creating
  a new window, doing nothing, showing a window that disappears
  inmediately or even crash.

  Tested with:
  Arch Linux with all packages updated (2016/7/26)
  TWM as window manager
  QEMU (both stable 2.6.0-1 and latest git commit f49ee63)
  Command: qemu-system-x86_64 -display sdl

  sdl2 version 2.0.4-2

  How to reproduce:
  1. Open QEMU with the given command
  2. Try to switch console (Ctrl-Alt-2 for example)

  Expected behaviour:
  As in GTK, the window should now show the desired console

  Actual behaviour:
  Here I have to say I can't explain it very well. Almost always it just 
creates a new window that shows the desired console, but it is closed 
inmediatley. If not, it opens a new window that keeps open, and it sometimes is 
responsive, but further attempts to switch consoles end causing a crash, and 
QEMU has to be SIGKILLed

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1606708/+subscriptions

Re: [Qemu-devel] [PATCH v6 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190913172741.5662-1-mlevi...@redhat.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

libudev   no
default devices   yes

warning: Python 2 support is deprecated
warning: Python 3 will be required for building future versions of QEMU
cross containers  no

NOTE: guest cross-compilers enabled: cc


The full log is available at
http://patchew.org/logs/20190913172741.5662-1-mlevi...@redhat.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [Qemu-devel] [PATCH] migration: check length directly to make sure the range is aligned

[Wei Yang]

On Fri, Jul 19, 2019 at 06:54:00PM +0100, Dr. David Alan Gilbert wrote:
>* Wei Yang (richardw.y...@linux.intel.com) wrote:
>> Since the start addr is already checked, to make sure the range is
>> aligned, checking the length is enough.
>> 
>> Signed-off-by: Wei Yang 
>> ---
>>  exec.c | 7 +++
>>  1 file changed, 3 insertions(+), 4 deletions(-)
>> 
>> diff --git a/exec.c b/exec.c
>> index 50ea9c5aaa..8fa980baae 100644
>> --- a/exec.c
>> +++ b/exec.c
>> @@ -4067,10 +4067,9 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t 
>> start, size_t length)
>>  
>>  if ((start + length) <= rb->used_length) {
>>  bool need_madvise, need_fallocate;
>> -uint8_t *host_endaddr = host_startaddr + length;
>> -if ((uintptr_t)host_endaddr & (rb->page_size - 1)) {
>> -error_report("ram_block_discard_range: Unaligned end address: 
>> %p",
>> - host_endaddr);
>> +if (length & (rb->page_size - 1)) {
>> +error_report("ram_block_discard_range: Unaligned length: %lx",
>> + length);
>
>Yes, I *think* this is safe, we'll need to watch out for any warnings;
>David Gibson's balloon fix from February means that the balloon code
>should now warn in it's case.
>
>rth: Want to pick this up?
>
>Reviewed-by: Dr. David Alan Gilbert 
>

Any other task I should do for progress?

>>  goto err;
>>  }
>>  
>> -- 
>> 2.17.1
>> 
>--
>Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

-- 
Wei Yang
Help you, Help me

Re: [Qemu-devel] [PATCH] migration: use migration_is_active to represent active state

[Wei Yang]

On Wed, Jul 17, 2019 at 08:53:41AM +0800, Wei Yang wrote:
>Wrap the check into a function to make it easy to read.
>

Hi, Dave & Juan

Do you like this one :-) ?

>Signed-off-by: Wei Yang 
>---
> include/migration/misc.h |  1 +
> migration/migration.c| 12 
> 2 files changed, 9 insertions(+), 4 deletions(-)
>
>diff --git a/include/migration/misc.h b/include/migration/misc.h
>index 5cdbabd094..42d6abc920 100644
>--- a/include/migration/misc.h
>+++ b/include/migration/misc.h
>@@ -61,6 +61,7 @@ void migration_object_init(void);
> void migration_shutdown(void);
> void qemu_start_incoming_migration(const char *uri, Error **errp);
> bool migration_is_idle(void);
>+bool migration_is_active(MigrationState *);
> void add_migration_state_change_notifier(Notifier *notify);
> void remove_migration_state_change_notifier(Notifier *notify);
> bool migration_in_setup(MigrationState *);
>diff --git a/migration/migration.c b/migration/migration.c
>index 43fd8297ef..4c066fc85c 100644
>--- a/migration/migration.c
>+++ b/migration/migration.c
>@@ -1529,8 +1529,7 @@ static void migrate_fd_cleanup(MigrationState *s)
> qemu_fclose(tmp);
> }
> 
>-assert((s->state != MIGRATION_STATUS_ACTIVE) &&
>-   (s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE));
>+assert(!migration_is_active(s));
> 
> if (s->state == MIGRATION_STATUS_CANCELLING) {
> migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
>@@ -1690,6 +1689,12 @@ bool migration_is_idle(void)
> return false;
> }
> 
>+bool migration_is_active(MigrationState *s)
>+{
>+return (s->state == MIGRATION_STATUS_ACTIVE ||
>+s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
>+}
>+
> void migrate_init(MigrationState *s)
> {
> /*
>@@ -3226,8 +3231,7 @@ static void *migration_thread(void *opaque)
> 
> trace_migration_thread_setup_complete();
> 
>-while (s->state == MIGRATION_STATUS_ACTIVE ||
>-   s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
>+while (migration_is_active(s)) {
> int64_t current_time;
> 
> if (urgent || !qemu_file_rate_limit(s->to_dst_file)) {
>-- 
>2.17.1
>

-- 
Wei Yang
Help you, Help me

Re: [Qemu-devel] [PATCH v2 0/2] refine memory_device_get_free_addr

[Wei Yang]

On Tue, Jul 30, 2019 at 08:37:38AM +0800, Wei Yang wrote:
>When we iterate the memory-device list to get the available range, it is not
>necessary to iterate the whole list.
>
>1) no more overlap for hinted range if tmp exceed it
>
>v2:
>   * remove #2 as suggested by Igor and David
>   * add some comment to inform address assignment stay the same as before
> this change 
>
>Wei Yang (2):
>  memory-device: not necessary to use goto for the last check
>  memory-device: break the loop if tmp exceed the hinted range
>
> hw/mem/memory-device.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>

Would someone take this patch set?

>-- 
>2.17.1
>

-- 
Wei Yang
Help you, Help me

Re: [Qemu-devel] [PATCH 3/4] iotests: Add @error to wait_until_completed

[John Snow]

On 9/12/19 9:56 AM, Max Reitz wrote:
> Callers can use this new parameter to expect failure during the
> completion process.
> 
> Signed-off-by: Max Reitz 
> ---
>  tests/qemu-iotests/iotests.py | 18 --
>  1 file changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index b26271187c..300347c7c8 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -745,15 +745,20 @@ class QMPTestCase(unittest.TestCase):
>  self.assert_no_active_block_jobs()
>  return result
>  
> -def wait_until_completed(self, drive='drive0', check_offset=True, 
> wait=60.0):
> +def wait_until_completed(self, drive='drive0', check_offset=True, 
> wait=60.0,
> + error=None):
>  '''Wait for a block job to finish, returning the event'''
>  while True:
>  for event in self.vm.get_qmp_events(wait=wait):
>  if event['event'] == 'BLOCK_JOB_COMPLETED':
>  self.assert_qmp(event, 'data/device', drive)
> -self.assert_qmp_absent(event, 'data/error')
> -if check_offset:
> -self.assert_qmp(event, 'data/offset', 
> event['data']['len'])
> +if error is None:
> +self.assert_qmp_absent(event, 'data/error')
> +if check_offset:
> +self.assert_qmp(event, 'data/offset',
> +event['data']['len'])
> +else:
> +self.assert_qmp(event, 'data/error', error)
>  self.assert_no_active_block_jobs()
>  return event
>  elif event['event'] == 'JOB_STATUS_CHANGE':
> @@ -771,7 +776,8 @@ class QMPTestCase(unittest.TestCase):
>  self.assert_qmp(event, 'data/type', 'mirror')
>  self.assert_qmp(event, 'data/offset', event['data']['len'])
>  
> -def complete_and_wait(self, drive='drive0', wait_ready=True):
> +def complete_and_wait(self, drive='drive0', wait_ready=True,
> +  completion_error=None):
>  '''Complete a block job and wait for it to finish'''
>  if wait_ready:
>  self.wait_ready(drive=drive)
> @@ -779,7 +785,7 @@ class QMPTestCase(unittest.TestCase):
>  result = self.vm.qmp('block-job-complete', device=drive)
>  self.assert_qmp(result, 'return', {})
>  
> -event = self.wait_until_completed(drive=drive)
> +event = self.wait_until_completed(drive=drive, 
> error=completion_error)
>  self.assert_qmp(event, 'data/type', 'mirror')
>  
>  def pause_wait(self, job_id='job0'):
> 

toot toot more optional parameters. lay them at the altar of
noncommittal python design.

I completely forget what the difference between unittest.TestCase and
qtest.QEMUQtestMachine is and why they each have job management methods.

Well, OK: the VM one is a simple subclass of the general-purpose VM
machine to add some more useful stuff. the unittest one implements some
general-purpose behavior with asserts that only work in the unittest world.

Still,

It's a little fun that we've got run_job as well as cancel_and_wait,
wait_until_completed, wait_ready, wait_ready_and_cancel, pause_wait and
pause_job and they all seem to implement job run-state logic management
a little differently.

Probably no bugs there, I bet.

*cough* Not your fault, anyway, so please take this accolade:

Reviewed-by: John Snow 


(it's probably my fault)

Re: [Qemu-devel] [PATCH 1/4] mirror: Do not dereference invalid pointers

[John Snow]

On 9/12/19 9:56 AM, Max Reitz wrote:
> mirror_exit_common() may be called twice (if it is called from
> mirror_prepare() and fails, it will be called from mirror_abort()
> again).
> 
> In such a case, many of the pointers in the MirrorBlockJob object will
> already be freed.  This can be seen most reliably for s->target, which
> is set to NULL (and then dereferenced by blk_bs()).
> 
> Cc: qemu-sta...@nongnu.org
> Fixes: 737efc1eda23b904fbe0e66b37715fb0e5c3e58b
> Signed-off-by: Max Reitz 

Sorry that I left the mirror callbacks such a mess. I think I got the
design for the completion callbacks completely wrong, because of how
hard it is to disentangle mirror into something reasonable here.

The original idea was that it calls .prepare, then either .commit or
.abort, then .clean. Ideally, there would be no exit_common for mirror;
everything would be sorted into the proper little callback silos.

The problem with the existing design is that if it has already failed by
.prepare time, we jump straight to .abort, so it has this uneven,
unbalanced design. The code in abort and cleanup therefore has to work
doubly-hard to figure out what exactly it needs to do.

It's a mess.

I wonder if it can be improved by always calling prepare, even when
we've already failed. We could just posit that it now means "prepare to
commit" or "prepare to abort" but we can do all of the work that can
still fail there in one shot.

Maybe that will make the work that needs to happen in abort/commit
easier to digest.

> ---
>  block/mirror.c | 13 +
>  1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/block/mirror.c b/block/mirror.c
> index fe984efb90..706d80fced 100644
> --- a/block/mirror.c
> +++ b/block/mirror.c
> @@ -620,11 +620,11 @@ static int mirror_exit_common(Job *job)
>  {
>  MirrorBlockJob *s = container_of(job, MirrorBlockJob, common.job);
>  BlockJob *bjob = &s->common;
> -MirrorBDSOpaque *bs_opaque = s->mirror_top_bs->opaque;
> +MirrorBDSOpaque *bs_opaque;
>  AioContext *replace_aio_context = NULL;
> -BlockDriverState *src = s->mirror_top_bs->backing->bs;
> -BlockDriverState *target_bs = blk_bs(s->target);
> -BlockDriverState *mirror_top_bs = s->mirror_top_bs;
> +BlockDriverState *src;
> +BlockDriverState *target_bs;
> +BlockDriverState *mirror_top_bs;
>  Error *local_err = NULL;
>  bool abort = job->ret < 0;
>  int ret = 0;
> @@ -634,6 +634,11 @@ static int mirror_exit_common(Job *job)
>  }
>  s->prepared = true;
>  
> +mirror_top_bs = s->mirror_top_bs;
> +bs_opaque = mirror_top_bs->opaque;
> +src = mirror_top_bs->backing->bs;
> +target_bs = blk_bs(s->target);
> +
>  if (bdrv_chain_contains(src, target_bs)) {
>  bdrv_unfreeze_backing_chain(mirror_top_bs, target_bs);
>  }
> 

Reviewed-by: John Snow 

Re: [Qemu-devel] [PATCH v2 3/3] block/qcow2: proper locking on bitmap add/remove paths

[John Snow]

On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote:
> qmp_block_dirty_bitmap_add and do_block_dirty_bitmap_remove do acquire
> aio context since 0a6c86d024c52b. But this is not enough: we also must
> lock qcow2 mutex when access in-image metadata. Especially it concerns
> freeing qcow2 clusters.
> 
> To achieve this, move qcow2_can_store_new_dirty_bitmap and
> qcow2_remove_persistent_dirty_bitmap to coroutine context.
> 
> Since we work in coroutines in correct aio context, we don't need
> context acquiring in blockdev.c anymore, drop it.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

It makes sense to me given the context, but I'll want Kevin or Max to
eye it over for locking correctness, admittedly.

Here's my good faith RB:

Reviewed-by: John Snow 


(I'm not so sure about the aio_wait_kick. Why is that needed here? ...

oh, 4720cbeea1f42fd905fc69338fd42b191e58b412 explains why.
I suppose any synchronous caller of a co_ function needs to make sure
that the entry shim kicks on the way out. This seems very easy to get
wrong, doesn't it?)

Re: [Qemu-devel] [PATCH v3] qemu-ga: Convert invocation documentation to rST

[Michael Roth]

Quoting Peter Maydell (2019-09-13 08:28:36)
> On Thu, 5 Sep 2019 at 14:10, Peter Maydell  wrote:
> >
> > The qemu-ga documentation is currently in qemu-ga.texi in
> > Texinfo format, which we present to the user as:
> >  * a qemu-ga manpage
> >  * a section of the main qemu-doc HTML documentation
> >
> > Convert the documentation to rST format, and present it to
> > the user as:
> >  * a qemu-ga manpage
> >  * part of the interop/ Sphinx manual
> >
> > Signed-off-by: Peter Maydell 
> > Reviewed-by: Michael Roth 
> > Tested-by: Michael Roth 
> > ---
> > Now we're out of release we might as well get this in the tree.
> > I saw a new manpage in texi going past the other day so perhaps
> > if we have the framework for doing manpages in rst in-tree
> > we can avoid adding too many more files we need to convert later.
> 
> I'm planning to put this into a target-arm pullreq since
> I'm doing one anyway.

Ok, sounds good to me.

> 
> thanks
> -- PMM

Re: [Qemu-devel] [PATCH 1/1] qga-win: network-get-interfaces command name field bug fix

[Michael Roth]

Quoting Bishara AbuHattoum (2019-08-19 08:16:20)
> Network interface name is fetched as an encoded WCHAR array, (wide
> character), then it is decoded using the guest's CP_ACP Windows code
> page, which is the default code page as configure in the guest's
> Windows, then it is returned as a byte array, (char array).
> 
> As stated in the BZ#1733165, when renaming a network interface to a
> Chinese name and invoking this command, the returned name field has
> the (\ufffd) value for each Chinese character the name had, this
> value is an indication that the code page does not have the decoding
> information for the given character.
> 
> This bug is a result of using the CP_ACP code page for decoding which
> is an interchangeable code page, instead CP_UTF8 code page should be
> used for decoding the network interface's name.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1733165
> 
> Signed-off-by: Bishara AbuHattoum 

Thanks, applied to qga tree:
  https://github.com/mdroth/qemu/commits/qga

> ---
>  qga/commands-win32.c | 10 +-
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/qga/commands-win32.c b/qga/commands-win32.c
> index 6b67f16faf..64b1c754b0 100644
> --- a/qga/commands-win32.c
> +++ b/qga/commands-win32.c
> @@ -1387,12 +1387,12 @@ static IP_ADAPTER_ADDRESSES 
> *guest_get_adapters_addresses(Error **errp)
>  static char *guest_wctomb_dup(WCHAR *wstr)
>  {
>  char *str;
> -size_t i;
> +size_t str_size;
> 
> -i = wcslen(wstr) + 1;
> -str = g_malloc(i);
> -WideCharToMultiByte(CP_ACP, WC_COMPOSITECHECK,
> -wstr, -1, str, i, NULL, NULL);
> +str_size = WideCharToMultiByte(CP_UTF8, 0, wstr, -1, NULL, 0, NULL, 
> NULL);
> +/* add 1 to str_size for NULL terminator */
> +str = g_malloc(str_size + 1);
> +WideCharToMultiByte(CP_UTF8, 0, wstr, -1, str, str_size, NULL, NULL);
>  return str;
>  }
> 
> -- 
> 2.17.2
> 

Re: [Qemu-devel] [PATCH v2 2/3] block/dirty-bitmap: return int from bdrv_remove_persistent_dirty_bitmap

[John Snow]

On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote:
> It's more comfortable to not deal with local_err.
> 

I agree.

> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  block/qcow2.h|  5 ++---
>  include/block/block_int.h|  6 +++---
>  include/block/dirty-bitmap.h |  5 ++---
>  block/dirty-bitmap.c |  9 +
>  block/qcow2-bitmap.c | 20 +++-
>  blockdev.c   |  7 +++
>  6 files changed, 26 insertions(+), 26 deletions(-)
> 
> diff --git a/block/qcow2.h b/block/qcow2.h
> index 998bcdaef1..99ee88f802 100644
> --- a/block/qcow2.h
> +++ b/block/qcow2.h
> @@ -747,9 +747,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState 
> *bs,
>const char *name,
>uint32_t granularity,
>Error **errp);
> -void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
> -  const char *name,
> -  Error **errp);
> +int qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
> *name,
> + Error **errp);
>  
>  ssize_t coroutine_fn
>  qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 0422acdf1c..503ac9e3cd 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -556,9 +556,9 @@ struct BlockDriver {
>  const char *name,
>  uint32_t granularity,
>  Error **errp);
> -void (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
> -const char *name,
> -Error **errp);
> +int (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
> +   const char *name,
> +   Error **errp);
>  
>  /**
>   * Register/unregister a buffer for I/O. For example, when the driver is
> diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
> index 4b4b731b46..07503b03b5 100644
> --- a/include/block/dirty-bitmap.h
> +++ b/include/block/dirty-bitmap.h
> @@ -37,9 +37,8 @@ int bdrv_dirty_bitmap_check(const BdrvDirtyBitmap *bitmap, 
> uint32_t flags,
>  Error **errp);
>  void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap 
> *bitmap);
>  void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs);
> -void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
> - const char *name,
> - Error **errp);
> +int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
> *name,
> +Error **errp);
>  void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
>  void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
>  void bdrv_enable_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap);
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 8f42015db9..a52b83b619 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -455,13 +455,14 @@ void bdrv_release_named_dirty_bitmaps(BlockDriverState 
> *bs)
>   * not fail.
>   * This function doesn't release corresponding BdrvDirtyBitmap.
>   */
> -void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
> - const char *name,
> - Error **errp)
> +int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
> *name,
> +Error **errp)
>  {
>  if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) {
> -bs->drv->bdrv_remove_persistent_dirty_bitmap(bs, name, errp);
> +return bs->drv->bdrv_remove_persistent_dirty_bitmap(bs, name, errp);
>  }
> +

But is it a problem if we return an error code without setting errp now?
If this is for the sake of not having to deal with local_err, we should
make sure that a non-zero return means that errp is set. Right?

> +return -ENOTSUP;
>  }
>  
>  bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
> diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
> index b2487101ed..1aaedb3b55 100644
> --- a/block/qcow2-bitmap.c
> +++ b/block/qcow2-bitmap.c
> @@ -1404,11 +1404,10 @@ static Qcow2Bitmap 
> *find_bitmap_by_name(Qcow2BitmapList *bm_list,
>  return NULL;
>  }
>  
> -void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
> -  const char *name,
> -  Error **errp)
> +int qcow2_remove_persistent_dirty_bitmap(

Re: [Qemu-devel] [PATCH v2 1/3] block: move bdrv_can_store_new_dirty_bitmap to block/dirty-bitmap.c

[John Snow]

On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote:
> block/dirty-bitmap.c seems to be more appropriate for it and
> bdrv_remove_persistent_dirty_bitmap already in it.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

No disagreements here. I think we put it in block.c initially because
it's a bdrv callback/dispatch function, but dirty-block.c is just a
logical subdivision of block.c, so it's fine.

Reviewed-by: John Snow 

Re: [Qemu-devel] [PATCH] podman: fix command invocation

[Alex Bennée]

John Snow  writes:

> Oops; there's no argv here.

Oops indeed. Queued to testing/next, thanks.

>
> Signed-off-by: John Snow 
> ---
>  tests/docker/docker.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 29613afd48..bc7a470ca2 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -334,7 +334,7 @@ class Docker(object):
>  cmd = [ "-u", str(uid) ] + cmd
>  # podman requires a bit more fiddling
>  if self._command[0] == "podman":
> -argv.insert(0, '--userns=keep-id')
> +cmd.insert(0, '--userns=keep-id')
>
>  ret = self._do_check(["run", "--label",
>   "com.qemu.instance.uuid=" + label] + cmd,


--
Alex Bennée

Re: [Qemu-devel] Python3 support for patches

[John Snow]

On 9/13/19 3:32 PM, John Snow wrote:
> Hi,
> 
> I quite like the patches tool; but python2 is notably doomed.
> 
> I tried my hand at polishing it up for python3 and pushed the results
> here: https://github.com/jnsnow/patches/tree/python3
> 
> I didn't faff around with trying to add simultaneous support: this is a
> direct conversion in one big chunk. I used 2to3 and supplemented with
> pylint and flake8 to find problem spots.
> 
> I don't use all of the features of this tool, but maybe if you'd like to
> give this a try you can test this branch and let me know if I missed any
> spots.
> 
> You can install patches 0.3.1 using the standard incantations:
>> python3 setup.py install --user
> 
> If it's too broken, you can uninstall it later with:
>> pip3 uninstall patches
> 
> You may need to check back to the master branch and force a
> reinstallation of the python2 version to get the bin back in your PATH:
> 
>> git checkout master
>> python2 setup.py install --user
> 
> 
> Happy python2 doomsday,
> --js
> 

Ah, this was definitely premature. I had a dream last night that this
was working just fine, but in the harsh, unforgiving sunlight, there are
still a few problems.

Needs a few more minutes in the oven.

--js

Re: [Qemu-devel] [PATCH v1] gdbstub: riscv: fix the fflags registers

[Palmer Dabbelt]

On Tue, 10 Sep 2019 01:15:41 PDT (-0700), frederic.kon...@adacore.com wrote:

While debugging an application with GDB the following might happen:

(gdb) return
Make xxx return now? (y or n) y
Could not fetch register "fflags"; remote failure reply 'E14'

This is because riscv_gdb_get_fpu calls riscv_csrrw_debug with a wrong csr
number (8). It should use the csr_register_map in order to reach the
riscv_cpu_get_fflags callback.

Signed-off-by: KONRAD Frederic 
---
 target/riscv/gdbstub.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/target/riscv/gdbstub.c b/target/riscv/gdbstub.c
index 27be932..ded140e 100644
--- a/target/riscv/gdbstub.c
+++ b/target/riscv/gdbstub.c
@@ -313,7 +313,8 @@ static int riscv_gdb_get_fpu(CPURISCVState *env, uint8_t 
*mem_buf, int n)
  * register 33, so we recalculate the map index.
  * This also works for CSR_FRM and CSR_FCSR.
  */
-result = riscv_csrrw_debug(env, n - 33 +  8, &val, 0, 0);
+result = riscv_csrrw_debug(env, n - 33 + csr_register_map[8], &val,
+   0, 0);
 if (result == 0) {
 return gdb_get_regl(mem_buf, val);
 }
@@ -335,7 +336,8 @@ static int riscv_gdb_set_fpu(CPURISCVState *env, uint8_t 
*mem_buf, int n)
  * register 33, so we recalculate the map index.
  * This also works for CSR_FRM and CSR_FCSR.
  */
-result = riscv_csrrw_debug(env, n - 33 + 8, NULL, val, -1);
+result = riscv_csrrw_debug(env, n - 33 + csr_register_map[8], NULL,
+   val, -1);
 if (result == 0) {
 return sizeof(target_ulong);
 }


Reviewed-by: Palmer Dabbelt 

I just tagged a fixed version of my PR, but I'll include this in the next one.

Re: [Qemu-devel] [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

[John Snow]

On 9/13/19 5:00 PM, Philippe Mathieu-Daudé wrote:
> On 9/13/19 10:01 PM, John Snow wrote:
>> On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote:
>>> Event format ending with newlines confuse the trace reports.
>>> Forbid them.
>>>
>>> Add a check to refuse new format added with trailing newline:
>>>
>>>   $ make
>>>   [...]
>>> GEN hw/misc/trace.h
>>>   Traceback (most recent call last):
>>> File "scripts/tracetool.py", line 152, in 
>>>   main(sys.argv)
>>> File "scripts/tracetool.py", line 143, in main
>>>   events.extend(tracetool.read_events(fh, arg))
>>> File "scripts/tracetool/__init__.py", line 367, in read_events
>>>   event = Event.build(line)
>>> File "scripts/tracetool/__init__.py", line 281, in build
>>>   raise ValueError("Event format can not end with a newline character")
>>>   ValueError: Error at hw/misc/trace-events:121: Event format can not end 
>>> with a newline character
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé 
>>> ---
>>>  docs/devel/tracing.txt| 2 ++
>>>  scripts/tracetool/__init__.py | 3 +++
>>>  2 files changed, 5 insertions(+)
>>>
>>> diff --git a/docs/devel/tracing.txt b/docs/devel/tracing.txt
>>> index 76e492a489..8231bbf5d1 100644
>>> --- a/docs/devel/tracing.txt
>>> +++ b/docs/devel/tracing.txt
>>> @@ -112,6 +112,8 @@ Trace events should use types as follows:
>>>  Format strings should reflect the types defined in the trace event.  Take
>>>  special care to use PRId64 and PRIu64 for int64_t and uint64_t types,
>>>  respectively.  This ensures portability between 32- and 64-bit platforms.
>>> +Format strings must not end with a newline character.  It is the 
>>> responsibility
>>> +of backends to adapt line ending for proper logging.
>>>  
>>>  Each event declaration will start with the event name, then its arguments,
>>>  finally a format string for pretty-printing. For example:
>>> diff --git a/scripts/tracetool/__init__.py b/scripts/tracetool/__init__.py
>>> index 6fca674936..57df74e67c 100644
>>> --- a/scripts/tracetool/__init__.py
>>> +++ b/scripts/tracetool/__init__.py
>>> @@ -277,6 +277,9 @@ class Event(object):
>>>  if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
>>>  raise ValueError("Event format '%m' is forbidden, pass the 
>>> error "
>>>   "as an explicit trace argument")
>>> +if fmt.endswith("\\n\""):
>>> +raise ValueError("Event format must not end with a newline "
>>
>> It's barely worth mentioning, but you can use r"\n" for cases like this,
>> if it makes it easier to read.
> 
> TIL Python r"" :)
> 
> This would be r"\n\"", right? We need to match the trailing '"'.
> Same length, not sure which string is easier to review =)
> 

Ah, wow, I did misread this. It's confusing!

In this case, you can use r'\n"' to get the job done.

(My r-b stands, anyway.)

--js

Re: [Qemu-devel] [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

[Philippe Mathieu-Daudé]

On 9/13/19 10:01 PM, John Snow wrote:
> On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote:
>> Event format ending with newlines confuse the trace reports.
>> Forbid them.
>>
>> Add a check to refuse new format added with trailing newline:
>>
>>   $ make
>>   [...]
>> GEN hw/misc/trace.h
>>   Traceback (most recent call last):
>> File "scripts/tracetool.py", line 152, in 
>>   main(sys.argv)
>> File "scripts/tracetool.py", line 143, in main
>>   events.extend(tracetool.read_events(fh, arg))
>> File "scripts/tracetool/__init__.py", line 367, in read_events
>>   event = Event.build(line)
>> File "scripts/tracetool/__init__.py", line 281, in build
>>   raise ValueError("Event format can not end with a newline character")
>>   ValueError: Error at hw/misc/trace-events:121: Event format can not end 
>> with a newline character
>>
>> Signed-off-by: Philippe Mathieu-Daudé 
>> ---
>>  docs/devel/tracing.txt| 2 ++
>>  scripts/tracetool/__init__.py | 3 +++
>>  2 files changed, 5 insertions(+)
>>
>> diff --git a/docs/devel/tracing.txt b/docs/devel/tracing.txt
>> index 76e492a489..8231bbf5d1 100644
>> --- a/docs/devel/tracing.txt
>> +++ b/docs/devel/tracing.txt
>> @@ -112,6 +112,8 @@ Trace events should use types as follows:
>>  Format strings should reflect the types defined in the trace event.  Take
>>  special care to use PRId64 and PRIu64 for int64_t and uint64_t types,
>>  respectively.  This ensures portability between 32- and 64-bit platforms.
>> +Format strings must not end with a newline character.  It is the 
>> responsibility
>> +of backends to adapt line ending for proper logging.
>>  
>>  Each event declaration will start with the event name, then its arguments,
>>  finally a format string for pretty-printing. For example:
>> diff --git a/scripts/tracetool/__init__.py b/scripts/tracetool/__init__.py
>> index 6fca674936..57df74e67c 100644
>> --- a/scripts/tracetool/__init__.py
>> +++ b/scripts/tracetool/__init__.py
>> @@ -277,6 +277,9 @@ class Event(object):
>>  if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
>>  raise ValueError("Event format '%m' is forbidden, pass the 
>> error "
>>   "as an explicit trace argument")
>> +if fmt.endswith("\\n\""):
>> +raise ValueError("Event format must not end with a newline "
> 
> It's barely worth mentioning, but you can use r"\n" for cases like this,
> if it makes it easier to read.

TIL Python r"" :)

This would be r"\n\"", right? We need to match the trailing '"'.
Same length, not sure which string is easier to review =)

[Qemu-devel] [PATCH] error-report: Add info_report_once helper

[Cyrill Gorcunov]

We already have error_report_once and warn_report_once,
thus lets add info_report_once to complement. Actually
I use this helper a lot so might be usefull for others.

Signed-off-by: Cyrill Gorcunov 
---
 include/qemu/error-report.h |   13 +
 util/qemu-error.c   |   20 
 2 files changed, 33 insertions(+)

Index: vanilla.git/include/qemu/error-report.h
===
--- vanilla.git.orig/include/qemu/error-report.h
+++ vanilla.git/include/qemu/error-report.h
@@ -47,6 +47,8 @@ bool error_report_once_cond(bool *printe
 GCC_FMT_ATTR(2, 3);
 bool warn_report_once_cond(bool *printed, const char *fmt, ...)
 GCC_FMT_ATTR(2, 3);
+bool info_report_once_cond(bool *printed, const char *fmt, ...)
+GCC_FMT_ATTR(2, 3);
 
 void error_init(const char *argv0);
 
@@ -72,6 +74,17 @@ void error_init(const char *argv0);
   fmt, ##__VA_ARGS__);  \
 })
 
+/*
+ * Similar to info_report(), except it prints the message just once.
+ * Return true when it prints, false otherwise.
+ */
+#define info_report_once(fmt, ...)  \
+({  \
+static bool print_once_;\
+info_report_once_cond(&print_once_, \
+  fmt, ##__VA_ARGS__);  \
+})
+
 const char *error_get_progname(void);
 extern bool enable_timestamp_msg;
 
Index: vanilla.git/util/qemu-error.c
===
--- vanilla.git.orig/util/qemu-error.c
+++ vanilla.git/util/qemu-error.c
@@ -350,6 +350,26 @@ bool warn_report_once_cond(bool *printed
 return true;
 }
 
+/*
+ * Like info_report(), except print just once.
+ * If *printed is false, print the message, and flip *printed to true.
+ * Return whether the message was printed.
+ */
+bool info_report_once_cond(bool *printed, const char *fmt, ...)
+{
+va_list ap;
+
+assert(printed);
+if (*printed) {
+return false;
+}
+*printed = true;
+va_start(ap, fmt);
+vreport(REPORT_TYPE_INFO, fmt, ap);
+va_end(ap);
+return true;
+}
+
 static char *qemu_glog_domains;
 
 static void qemu_log_func(const gchar *log_domain,

[Qemu-devel] [PATCH v3 12/16] docs/devel/qapi-code-gen: Reorder sections for readability

[Markus Armbruster]

Section "QMP/Guest agent schema" starts with a brief introduction,
then subsection "Comments", then subsection "Schema overview" (more
elaborate introduction), and only then talks about schema entities
like types, commands, and so forth.

Subsection "Comments" is long and tiring: almost 500 words, mostly
about doc comments.  Move the doc comment part to its own subsection
"Documentation comments" at the very end of "QMP/Guest agent schema".

Subsection "Schema overview" explains naming rules at considerable
length: 250 words.  Move this part to its own subsection "Naming rules
and reserved names" right after the subsections on schema entities.

Subsection "Enumeration types" is wedged between "Struct types" and
"Union types".  Move it before "Struct types".

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt | 436 ++-
 1 file changed, 221 insertions(+), 215 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index ec2d374483..663ef10a56 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -50,147 +50,6 @@ schema requires the use of JSON numbers or null.
 Comments are allowed; anything between an unquoted # and the following
 newline is ignored.
 
-A multi-line comment that starts and ends with a '##' line is a
-documentation comment.  These are parsed by the documentation
-generator, which recognizes certain markup detailed below.
-
-
- Documentation markup 
-
-Comment text starting with '=' is a section title:
-
-# = Section title
-
-Double the '=' for a subsection title:
-
-# == Subsection title
-
-'|' denotes examples:
-
-# | Text of the example, may span
-# | multiple lines
-
-'*' starts an itemized list:
-
-# * First item, may span
-#   multiple lines
-# * Second item
-
-You can also use '-' instead of '*'.
-
-A decimal number followed by '.' starts a numbered list:
-
-# 1. First item, may span
-#multiple lines
-# 2. Second item
-
-The actual number doesn't matter.  You could even use '*' instead of
-'2.' for the second item.
-
-Lists can't be nested.  Blank lines are currently not supported within
-lists.
-
-Additional whitespace between the initial '#' and the comment text is
-permitted.
-
-*foo* and _foo_ are for strong and emphasis styles respectively (they
-do not work over multiple lines). @foo is used to reference a name in
-the schema.
-
-Example:
-
-##
-# = Section
-# == Subsection
-#
-# Some text foo with *strong* and _emphasis_
-# 1. with a list
-# 2. like that
-#
-# And some code:
-# | $ echo foo
-# | -> do this
-# | <- get that
-#
-##
-
-
- Expression documentation 
-
-Expressions other than include and pragma directives may be preceded
-by a documentation block.  Such blocks are called expression
-documentation blocks.
-
-When documentation is required (see pragma 'doc-required'), expression
-documentation blocks are mandatory.
-
-The documentation block consists of a first line naming the
-expression, an optional overview, a description of each argument (for
-commands and events) or member (for structs, unions and alternates),
-and optional tagged sections.
-
-FIXME: the parser accepts these things in almost any order.
-
-Extensions added after the expression was first released carry a
-'(since x.y.z)' comment.
-
-A tagged section starts with one of the following words:
-"Note:"/"Notes:", "Since:", "Example"/"Examples", "Returns:", "TODO:".
-The section ends with the start of a new section.
-
-A 'Since: x.y.z' tagged section lists the release that introduced the
-expression.
-
-For example:
-
-##
-# @BlockStats:
-#
-# Statistics of a virtual block device or a block backing device.
-#
-# @device: If the stats are for a virtual block device, the name
-#  corresponding to the virtual block device.
-#
-# @node-name: The node name of the device. (since 2.3)
-#
-# ... more members ...
-#
-# Since: 0.14.0
-##
-{ 'struct': 'BlockStats',
-  'data': {'*device': 'str', '*node-name': 'str',
-   ... more members ... } }
-
-##
-# @query-blockstats:
-#
-# Query the @BlockStats for all virtual block devices.
-#
-# @query-nodes: If true, the command will query all the
-#   block nodes ... explain, explain ...  (since 2.3)
-#
-# Returns: A list of @BlockStats for each virtual block devices.
-#
-# Since: 0.14.0
-#
-# Example:
-#
-# -> { "execute": "query-blockstats" }
-# <- {
-#  ... lots of output ...
-#}
-#
-##
-{ 'command': 'query-blockstats',
-  'data': { '*query-nodes': 'bool' },
-  'returns': ['BlockStats'] }
-
- Free-form documentation 
-
-A documentation block that isn't an expression documentation block is
-a free-form documentation block.  These may be used to provide
-additional text and structuring content.
-
 
 === Schema overview ===
 
@@ -217,43 +76,6 @@ refers to a single-dimension array of that type; 
multi-dimension
 arrays are not directly supported (

[Qemu-devel] [PATCH v3 16/16] qapi: Tweak code to match docs/devel/qapi-code-gen.txt

[Markus Armbruster]

The previous commit made qapi-code-gen.txt define "(top-level)
expression" as either "directive" or "definition".  The code still
uses "expression" when it really means "definition".  Tidy up.

The previous commit made qapi-code-gen.txt use "object" rather than
"dictionary".  The code still uses "dictionary".  Tidy up.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py| 24 +--
 tests/qapi-schema/args-invalid.err|  2 +-
 tests/qapi-schema/doc-missing.err |  2 +-
 tests/qapi-schema/doc-no-symbol.err   |  2 +-
 .../qapi-schema/enum-dict-member-unknown.err  |  2 +-
 tests/qapi-schema/pragma-non-dict.err |  2 +-
 tests/qapi-schema/struct-data-invalid.err |  2 +-
 7 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index a538d2f37c..f27860540b 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -103,11 +103,11 @@ class QAPISemError(QAPIError):
 
 class QAPIDoc(object):
 """
-A documentation comment block, either expression or free-form
+A documentation comment block, either definition or free-form
 
-Expression documentation blocks consist of
+Definition documentation blocks consist of
 
-* a body section: one line naming the expression, followed by an
+* a body section: one line naming the definition, followed by an
   overview (any number of lines)
 
 * argument sections: a description of each argument (for commands
@@ -200,9 +200,9 @@ class QAPIDoc(object):
 Process a line of documentation text in the body section.
 
 If this a symbol line and it is the section's first line, this
-is an expression documentation block for that symbol.
+is a definition documentation block for that symbol.
 
-If it's an expression documentation block, another symbol line
+If it's a definition documentation block, another symbol line
 begins the argument section for the argument named by it, and
 a section tag begins an additional section.  Start that
 section and append the line to it.
@@ -220,7 +220,7 @@ class QAPIDoc(object):
 if not self.symbol:
 raise QAPIParseError(self._parser, "Invalid name")
 elif self.symbol:
-# This is an expression documentation block
+# This is a definition documentation block
 if name.startswith('@') and name.endswith(':'):
 self._append_line = self._append_args_line
 self._append_args_line(line)
@@ -428,7 +428,7 @@ class QAPISchemaParser(object):
 pragma = expr['pragma']
 if not isinstance(pragma, dict):
 raise QAPISemError(
-info, "Value of 'pragma' must be a dictionary")
+info, "Value of 'pragma' must be an object")
 for name, value in pragma.items():
 self._pragma(name, value, info)
 else:
@@ -437,7 +437,7 @@ class QAPISchemaParser(object):
 if cur_doc:
 if not cur_doc.symbol:
 raise QAPISemError(
-cur_doc.info, "Expression documentation required")
+cur_doc.info, "Definition documentation required")
 expr_elem['doc'] = cur_doc
 self.exprs.append(expr_elem)
 cur_doc = None
@@ -789,7 +789,7 @@ def check_type(info, source, value,
 
 if not isinstance(value, OrderedDict):
 raise QAPISemError(info,
-   "%s should be a dictionary or type name" % source)
+   "%s should be an object or type name" % source)
 
 # value is a dictionary, check that each member is okay
 for (key, arg) in value.items():
@@ -971,8 +971,8 @@ def check_enum(expr, info):
"Enum '%s' requires a string for 'prefix'" % name)
 
 for member in members:
-source = "dictionary member of enum '%s'" % name
-check_known_keys(info, source, member, ['name'], ['if'])
+check_known_keys(info, "member of enum '%s'" % name, member,
+ ['name'], ['if'])
 check_if(member, info)
 check_name(info, "Member of enum '%s'" % name, member['name'],
enum_member=True)
@@ -1081,7 +1081,7 @@ def check_exprs(exprs):
 
 if not doc and doc_required:
 raise QAPISemError(info,
-   "Expression missing documentation comment")
+   "Definition missing documentation comment")
 
 if 'enum' in expr:
 meta = 'enum'
diff --git a/tests/qapi-schema/args-invalid.err 
b/tests/qapi-schema/args-invalid.err
index fe1e94975b..bfb2e4133e 100644
--- a/tests/qapi-schema/args-invalid.err
+++ b/tests/qapi-s

Re: [Qemu-devel] [Qemu-block] [PATCH v2] virtio-blk: schedule virtio_notify_config to run on main context

[John Snow]

On 9/13/19 6:56 AM, Sergio Lopez wrote:
> virtio_notify_config() needs to acquire the global mutex, which isn't
> allowed from an iothread, and may lead to a deadlock like this:
> 
>  - main thead
>   * Has acquired: qemu_global_mutex.
>   * Is trying the acquire: iothread AioContext lock via
> AIO_WAIT_WHILE (after aio_poll).
> 
>  - iothread
>   * Has acquired: AioContext lock.
>   * Is trying to acquire: qemu_global_mutex (via
> virtio_notify_config->prepare_mmio_access).
> 
> If virtio_blk_resize() is called from an iothread, schedule
> virtio_notify_config() to be run in the main context BH.
> 
> Signed-off-by: Sergio Lopez 
> ---
> Changelog
> 
> v2:
>  - Use aio_bh_schedule_oneshot instead of scheduling a coroutine
>(thanks Kevin Wolf).
>  - Switch from RFC to v2 patch.
> ---
>  hw/block/virtio-blk.c | 21 -
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index 18851601cb..669dc60f5b 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -16,6 +16,7 @@
>  #include "qemu/iov.h"
>  #include "qemu/module.h"
>  #include "qemu/error-report.h"
> +#include "qemu/main-loop.h"
>  #include "trace.h"
>  #include "hw/block/block.h"
>  #include "hw/qdev-properties.h"
> @@ -1086,11 +1087,29 @@ static int virtio_blk_load_device(VirtIODevice *vdev, 
> QEMUFile *f,
>  return 0;
>  }
>  
> +static void virtio_resize_cb(void *opaque)
> +{
> +VirtIODevice *vdev = opaque;
> +
> +assert(qemu_get_current_aio_context() == qemu_get_aio_context());
> +virtio_notify_config(vdev);
> +}
> +
>  static void virtio_blk_resize(void *opaque)
>  {
>  VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
>  
> -virtio_notify_config(vdev);
> +if (qemu_get_current_aio_context() != qemu_get_aio_context()) {
> +/*
> + * virtio_notify_config() needs to acquire the global mutex,
> + * so it can't be called from an iothread. Instead, schedule
> + * it to be run in the main context BH.
> + */
> +aio_bh_schedule_oneshot(qemu_get_aio_context(),
> +virtio_resize_cb, vdev);
> +} else {
> +virtio_notify_config(vdev);
> +}
>  }
>  
>  static const BlockDevOps virtio_block_ops = {
> 

Do we need to bother to check our current context before firing off the
oneshot? It's simpler to just fire off the oneshot. I can't imagine that
resize is a hot path that needs to worry about being that efficient.

Well, I guess it doesn't hurt either.

Reviewed-by: John Snow 

[Qemu-devel] [PATCH v3 09/16] qapi: Permit alternates with just one branch

[Markus Armbruster]

A union or alternate without branches makes no sense and doesn't work:
it can't be instantiated.  A union or alternate with just one branch
works, but is degenerate.  We accept the former, but reject the
latter.  Weird.  docs/devel/qapi-code-gen.txt doesn't mention the
difference.  It claims an alternate definition is "is similar to a
simple union type".

Permit degenerate alternates to make them consistent with unions.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py  | 6 ++
 tests/qapi-schema/alternate-empty.err   | 2 +-
 tests/qapi-schema/alternate-empty.json  | 4 ++--
 tests/qapi-schema/qapi-schema-test.json | 4 +++-
 tests/qapi-schema/qapi-schema-test.out  | 6 --
 5 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index c5c71287c3..99db18f3d6 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -920,11 +920,9 @@ def check_alternate(expr, info):
 members = expr['data']
 types_seen = {}
 
-# Check every branch; require at least two branches
-if len(members) < 2:
+if len(members) == 0:
 raise QAPISemError(info,
-   "Alternate '%s' should have at least two branches "
-   "in 'data'" % name)
+   "Alternate '%s' cannot have empty 'data'" % name)
 for (key, value) in members.items():
 check_name(info, "Member of alternate '%s'" % name, key)
 check_known_keys(info,
diff --git a/tests/qapi-schema/alternate-empty.err 
b/tests/qapi-schema/alternate-empty.err
index bb06c5bfec..86dbc666eb 100644
--- a/tests/qapi-schema/alternate-empty.err
+++ b/tests/qapi-schema/alternate-empty.err
@@ -1 +1 @@
-tests/qapi-schema/alternate-empty.json:2: Alternate 'Alt' should have at least 
two branches in 'data'
+tests/qapi-schema/alternate-empty.json:2: Alternate 'Alt' cannot have empty 
'data'
diff --git a/tests/qapi-schema/alternate-empty.json 
b/tests/qapi-schema/alternate-empty.json
index fff15baf16..9f445474e6 100644
--- a/tests/qapi-schema/alternate-empty.json
+++ b/tests/qapi-schema/alternate-empty.json
@@ -1,2 +1,2 @@
-# alternates must list at least two types to be useful
-{ 'alternate': 'Alt', 'data': { 'i': 'int' } }
+# alternates cannot be empty
+{ 'alternate': 'Alt', 'data': { } }
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index e6dbbbd328..8b0d47c4ab 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -186,19 +186,21 @@
 
 # test that we correctly compile downstream extensions, as well as munge
 # ticklish names
+# also test union and alternate with just one branch
 { 'enum': '__org.qemu_x-Enum', 'data': [ '__org.qemu_x-value' ] }
 { 'struct': '__org.qemu_x-Base',
   'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }
 { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
   'data': { '__org.qemu_x-member2': 'str', '*wchar-t': 'int' } }
 { 'union': '__org.qemu_x-Union1', 'data': { '__org.qemu_x-branch': 'str' } }
+{ 'alternate': '__org.qemu_x-Alt1', 'data': { '__org.qemu_x-branch': 'str' } }
 { 'struct': '__org.qemu_x-Struct2',
   'data': { 'array': ['__org.qemu_x-Union1'] } }
 { 'union': '__org.qemu_x-Union2', 'base': '__org.qemu_x-Base',
   'discriminator': '__org.qemu_x-member1',
   'data': { '__org.qemu_x-value': '__org.qemu_x-Struct2' } }
 { 'alternate': '__org.qemu_x-Alt',
-  'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
+  'data': { '__org.qemu_x-branch': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
diff --git a/tests/qapi-schema/qapi-schema-test.out 
b/tests/qapi-schema/qapi-schema-test.out
index fb00a21996..bea7976bbb 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -274,6 +274,9 @@ object __org.qemu_x-Union1
 member type: __org.qemu_x-Union1Kind optional=False
 tag type
 case __org.qemu_x-branch: q_obj_str-wrapper
+alternate __org.qemu_x-Alt1
+tag type
+case __org.qemu_x-branch: str
 array __org.qemu_x-Union1List __org.qemu_x-Union1
 object __org.qemu_x-Struct2
 member array: __org.qemu_x-Union1List optional=False
@@ -283,8 +286,7 @@ object __org.qemu_x-Union2
 case __org.qemu_x-value: __org.qemu_x-Struct2
 alternate __org.qemu_x-Alt
 tag type
-case __org.qemu_x-branch: str
-case b: __org.qemu_x-Base
+case __org.qemu_x-branch: __org.qemu_x-Base
 event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
boxed=False
 array __org.qemu_x-EnumList __org.qemu_x-Enum
-- 
2.21.0

[Qemu-devel] [PATCH v3 10/16] qapi: Permit omitting all flat union branches

[Markus Armbruster]

Absent flat union branches default to the empty struct (since commit
800877bb16 "qapi: allow empty branches in flat unions").  But am
attempt to omit all of them is rejected with "Union 'FOO' has no
branches".  Harmless oddity, but it's easy to avoid, so do that.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt|  3 +--
 scripts/qapi/common.py  | 16 
 tests/qapi-schema/flat-union-empty.err  |  2 +-
 tests/qapi-schema/flat-union-empty.json |  2 +-
 tests/qapi-schema/qapi-schema-test.json |  5 +
 tests/qapi-schema/qapi-schema-test.out  |  9 +
 tests/qapi-schema/union-empty.err   |  2 +-
 tests/qapi-schema/union-empty.json  |  2 +-
 8 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 4ce67752a7..ec2d374483 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -436,8 +436,7 @@ Union types are used to let the user choose between several 
different
 variants for an object.  There are two flavors: simple (no
 discriminator or base), and flat (both discriminator and base).  A union
 type is defined using a data dictionary as explained in the following
-paragraphs.  The data dictionary for either type of union must not
-be empty.
+paragraphs.  Unions must have at least one branch.
 
 A simple union type defines a mapping from automatic discriminator
 values to data types like in this example:
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 99db18f3d6..3393a049cc 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -852,7 +852,7 @@ def check_union(expr, info):
 
 # With no discriminator it is a simple union.
 if discriminator is None:
-enum_define = None
+enum_values = members.keys()
 allow_metas = ['built-in', 'union', 'alternate', 'struct', 'enum']
 if base is not None:
 raise QAPISemError(info, "Simple union '%s' must not have a base" %
@@ -885,16 +885,17 @@ def check_union(expr, info):
'must not be conditional' %
(base, discriminator, name))
 enum_define = enum_types.get(discriminator_value['type'])
-allow_metas = ['struct']
 # Do not allow string discriminator
 if not enum_define:
 raise QAPISemError(info,
"Discriminator '%s' must be of enumeration "
"type" % discriminator)
+enum_values = enum_get_names(enum_define)
+allow_metas = ['struct']
+
+if (len(enum_values) == 0):
+raise QAPISemError(info, "Union '%s' has no branches" % name)
 
-# Check every branch; don't allow an empty union
-if len(members) == 0:
-raise QAPISemError(info, "Union '%s' cannot have empty 'data'" % name)
 for (key, value) in members.items():
 check_name(info, "Member of union '%s'" % name, key)
 
@@ -907,8 +908,8 @@ def check_union(expr, info):
 
 # If the discriminator names an enum type, then all members
 # of 'data' must also be members of the enum type.
-if enum_define:
-if key not in enum_get_names(enum_define):
+if discriminator is not None:
+if key not in enum_values:
 raise QAPISemError(info,
"Discriminator value '%s' is not found in "
"enum '%s'"
@@ -1578,7 +1579,6 @@ class QAPISchemaObjectTypeVariants(object):
 assert bool(tag_member) != bool(tag_name)
 assert (isinstance(tag_name, str) or
 isinstance(tag_member, QAPISchemaObjectTypeMember))
-assert len(variants) > 0
 for v in variants:
 assert isinstance(v, QAPISchemaObjectTypeVariant)
 self._tag_name = tag_name
diff --git a/tests/qapi-schema/flat-union-empty.err 
b/tests/qapi-schema/flat-union-empty.err
index 15754f54eb..fedbc0d1cf 100644
--- a/tests/qapi-schema/flat-union-empty.err
+++ b/tests/qapi-schema/flat-union-empty.err
@@ -1 +1 @@
-tests/qapi-schema/flat-union-empty.json:4: Union 'Union' cannot have empty 
'data'
+tests/qapi-schema/flat-union-empty.json:4: Union 'Union' has no branches
diff --git a/tests/qapi-schema/flat-union-empty.json 
b/tests/qapi-schema/flat-union-empty.json
index 77f1d9abfb..83e1cc7b96 100644
--- a/tests/qapi-schema/flat-union-empty.json
+++ b/tests/qapi-schema/flat-union-empty.json
@@ -1,4 +1,4 @@
-# flat unions cannot be empty
+# flat union discriminator cannot be empty
 { 'enum': 'Empty', 'data': [ ] }
 { 'struct': 'Base', 'data': { 'type': 'Empty' } }
 { 'union': 'Union', 'base': 'Base', 'discriminator': 'type', 'data': { } }
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index 8b0d47c4ab..75c42eb0e3 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi

[Qemu-devel] [PATCH v3 08/16] qapi: Permit 'boxed' with empty type

[Markus Armbruster]

We reject empty types with 'boxed': true.  We don't really need that
to work, but making it work is actually simpler than rejecting it, so
do that.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 tests/test-qmp-cmds.c   |  4 
 scripts/qapi/commands.py|  4 ++--
 scripts/qapi/common.py  | 14 ++
 scripts/qapi/events.py  | 10 +-
 tests/Makefile.include  |  1 -
 tests/qapi-schema/args-boxed-empty.err  |  1 -
 tests/qapi-schema/args-boxed-empty.exit |  1 -
 tests/qapi-schema/args-boxed-empty.json |  3 ---
 tests/qapi-schema/args-boxed-empty.out  |  0
 tests/qapi-schema/qapi-schema-test.json |  2 ++
 tests/qapi-schema/qapi-schema-test.out  |  4 
 11 files changed, 19 insertions(+), 25 deletions(-)
 delete mode 100644 tests/qapi-schema/args-boxed-empty.err
 delete mode 100644 tests/qapi-schema/args-boxed-empty.exit
 delete mode 100644 tests/qapi-schema/args-boxed-empty.json
 delete mode 100644 tests/qapi-schema/args-boxed-empty.out

diff --git a/tests/test-qmp-cmds.c b/tests/test-qmp-cmds.c
index ab389f42da..36fdf5b115 100644
--- a/tests/test-qmp-cmds.c
+++ b/tests/test-qmp-cmds.c
@@ -97,6 +97,10 @@ void qmp_boxed_union(UserDefListUnion *arg, Error **errp)
 {
 }
 
+void qmp_boxed_empty(Empty1 *arg, Error **errp)
+{
+}
+
 __org_qemu_x_Union1 *qmp___org_qemu_x_command(__org_qemu_x_EnumList *a,
   __org_qemu_x_StructList *b,
   __org_qemu_x_Union2 *c,
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index b929e07be4..7e3dd1068a 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -30,7 +30,7 @@ def gen_call(name, arg_type, boxed, ret_type):
 
 argstr = ''
 if boxed:
-assert arg_type and not arg_type.is_empty()
+assert arg_type
 argstr = '&arg, '
 elif arg_type:
 assert not arg_type.variants
@@ -96,7 +96,7 @@ def gen_marshal_decl(name):
 
 
 def gen_marshal(name, arg_type, boxed, ret_type):
-have_args = arg_type and not arg_type.is_empty()
+have_args = boxed or (arg_type and not arg_type.is_empty())
 
 ret = mcgen('''
 
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 0fb1d1956a..c5c71287c3 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -1687,12 +1687,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
 self.arg_type = schema.lookup_type(self._arg_type_name)
 assert isinstance(self.arg_type, QAPISchemaObjectType)
 self.arg_type.check(schema)
-if self.boxed:
-if self.arg_type.is_empty():
-raise QAPISemError(self.info,
-   "Cannot use 'boxed' with empty type")
-else:
-assert not self.arg_type.variants
+assert not self.arg_type.variants or self.boxed
 elif self.boxed:
 raise QAPISemError(self.info, "Use of 'boxed' requires 'data'")
 if self._ret_type_name:
@@ -1721,12 +1716,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 self.arg_type = schema.lookup_type(self._arg_type_name)
 assert isinstance(self.arg_type, QAPISchemaObjectType)
 self.arg_type.check(schema)
-if self.boxed:
-if self.arg_type.is_empty():
-raise QAPISemError(self.info,
-   "Cannot use 'boxed' with empty type")
-else:
-assert not self.arg_type.variants
+assert not self.arg_type.variants or self.boxed
 elif self.boxed:
 raise QAPISemError(self.info, "Use of 'boxed' requires 'data'")
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index b732581046..e0abfef7b0 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -65,6 +65,8 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, 
event_emit):
 # practice, we can rename our local variables with a leading _ prefix,
 # or split the code into a wrapper function that creates a boxed
 # 'param' object then calls another to do the real work.
+have_args = boxed or (arg_type and not arg_type.is_empty())
+
 ret = mcgen('''
 
 %(proto)s
@@ -73,15 +75,13 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, 
event_emit):
 ''',
 proto=build_event_send_proto(name, arg_type, boxed))
 
-if arg_type and not arg_type.is_empty():
+if have_args:
 ret += mcgen('''
 QObject *obj;
 Visitor *v;
 ''')
 if not boxed:
 ret += gen_param_var(arg_type)
-else:
-assert not boxed
 
 ret += mcgen('''
 
@@ -90,7 +90,7 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, 
event_emit):
 ''',
  name=name)
 
-if arg_type and not arg_type.is_empty():
+if have_args:
 ret += mcgen

[Qemu-devel] [PATCH v3 15/16] docs/devel/qapi-code-gen: Improve QAPI schema language doc

[Markus Armbruster]

We document the language by giving patterns of valid JSON objects.
The patterns contain placeholders we don't define anywhere; their
names have to speak for themselves.  I guess they do, but I'd prefer a
bit more rigor.  Provide a grammar instead, and rework the text
accordingly.

Documentation for QAPI schema conditionals (commit 967c885108,
6cc32b0e14, 87adbbffd4..3e270dcacc) and feature flags (commit
6a8c0b5102) was bolted on.  The sections documenting types, commands
and events don't mention them.  Section "Features" and "Configuring
the schema" then provide additional syntax for types, commands and
events.  I hate that.  Fix the sections documenting types, commands
and events to provide accurate syntax, and point to "Features" and
"Configuring the schema" for details.

We talk about "(top-level) expressions other than include and pragma".
Adopt more convenient terminology: a (top-level) expression is either
a directive (include or pragma) or a definition (anything else).

Avoid the terms "dictionary" and "key".  Stick to JSON terminology
"object" and "member name" instead.

While there, make spacing more consistent.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt | 596 ++-
 1 file changed, 372 insertions(+), 224 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index e75d680f6e..94f3054898 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -4,12 +4,12 @@ Copyright IBM Corp. 2011
 Copyright (C) 2012-2016 Red Hat, Inc.
 
 This work is licensed under the terms of the GNU GPL, version 2 or
-later. See the COPYING file in the top-level directory.
+later.  See the COPYING file in the top-level directory.
 
 == Introduction ==
 
 QAPI is a native C API within QEMU which provides management-level
-functionality to internal and external users. For external
+functionality to internal and external users.  For external
 users/processes, this interface is made available by a JSON-based wire
 format for the QEMU Monitor Protocol (QMP) for controlling qemu, as
 well as the QEMU Guest Agent (QGA) for communicating with the guest.
@@ -54,24 +54,49 @@ Differences:
 
 * Numbers are not supported.
 
-A QAPI schema consists of a series of top-level expressions (JSON
-objects).  Code and documentation is generated in schema definition
-order.  Code order should not matter.
+A second layer of syntax defines the sequences of JSON texts that are
+a correctly structured QAPI schema.  We provide a grammar for this
+syntax in an EBNF-like notation:
 
-The order of keys within JSON objects does not matter unless
+* Production rules look like non-terminal = expression
+* Concatenation: expression A B matches expression A, then B
+* Alternation: expression A | B matches expression A or B
+* Repetition: expression A... matches zero or more occurrences of
+  expression A
+* Repetition: expression A, ... matches zero or more occurrences of
+  expression A separated by ,
+* Grouping: expression ( A ) matches expression A
+* JSON's structural characters are terminals: { } [ ] : ,
+* JSON's literal names are terminals: false true null
+* String literals enclosed in 'single quotes' are terminal, and match
+  this JSON string, with a leading '*' stripped off
+* When JSON object member's name starts with '*', the member is
+  optional.
+* The symbol STRING is a terminal, and matches any JSON string
+* The symbol BOOL is a terminal, and matches JSON false or true
+* ALL-CAPS words other than STRING are non-terminals
+
+The order of members within JSON objects does not matter unless
 explicitly noted.
 
-There are eight kinds of top-level expressions: 'include', 'pragma',
-'command', 'struct', 'enum', 'union', 'alternate', and 'event'.  These
-are discussed in detail below.
-
-In the rest of this document, usage lines are given for each
-expression type, with literal strings written in lower case and
-placeholders written in capitals.  If a literal string includes a
-prefix of '*', that key/value pair can be omitted from the expression.
-For example, a usage statement that includes '*base':STRUCT-NAME
-means that an expression has an optional key 'base', which if present
-must have a value that forms a struct name.
+A QAPI schema consists of a series of top-level expressions:
+
+SCHEMA = TOP-LEVEL-EXPR...
+
+The top-level expressions are all JSON objects.  Code and
+documentation is generated in schema definition order.  Code order
+should not matter.
+
+A top-level expressions is either a directive or a definition:
+
+TOP-LEVEL-EXPR = DIRECTIVE | DEFINITION
+
+There are two kinds of directives and six kinds of definitions:
+
+DIRECTIVE = INCLUDE | PRAGMA
+DEFINITION = ENUM | STRUCT | UNION | ALTERNATE | COMMAND | EVENT
+
+These are discussed in detail below.
 
 
 === Built-in Types ===
@@ -101,16 +126,16 @@ The following types are predefined, and map to C as 
follows:
 
 === Include d

[Qemu-devel] [PATCH v3 03/16] qapi: Drop support for boxed alternate arguments

[Markus Armbruster]

Commands and events can define their argument type inline (default) or
by referring to another type ('boxed': true, since commit c818408e44
"qapi: Implement boxed types for commands/events", v2.7.0).  The
unboxed inline definition is an (anonymous) struct type.  The boxed
type may be a struct, union, or alternate type.

The latter is problematic: docs/interop/qemu-spec.txt requires the
value of the 'data' key to be a json-object, but any non-degenerate
alternate type has at least one branch that isn't.

Fortunately, we haven't made use of alternates in this context outside
tests/.  Drop support for them.

QAPISchemaAlternateType.is_empty() is now unused.  Drop it, too.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt| 12 ++--
 scripts/qapi/common.py  | 15 ---
 tests/qapi-schema/qapi-schema-test.json |  2 +-
 tests/qapi-schema/qapi-schema-test.out  |  2 +-
 4 files changed, 12 insertions(+), 19 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index e8ec8ac1de..3d3931fb7a 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -612,9 +612,9 @@ the command.  Normally, 'data' is a dictionary for an 
anonymous type,
 or names a struct type (possibly empty, but not a union), and its
 members are passed as separate arguments to this function.  If the
 command definition includes a key 'boxed' with the boolean value true,
-then 'data' is instead the name of any non-empty complex type
-(struct, union, or alternate), and a pointer to that QAPI type is
-passed as a single argument.
+then 'data' is instead the name of any non-empty complex type (struct
+or union), and a pointer to that QAPI type is passed as a single
+argument.
 
 The generator also emits a marshalling function that extracts
 arguments for the user's function out of an input QDict, calls the
@@ -714,9 +714,9 @@ The generator emits a function to send the event.  
Normally, 'data' is
 a dictionary for an anonymous type, or names a struct type (possibly
 empty, but not a union), and its members are passed as separate
 arguments to this function.  If the event definition includes a key
-'boxed' with the boolean value true, then 'data' is instead the name of
-any non-empty complex type (struct, union, or alternate), and a
-pointer to that QAPI type is passed as a single argument.
+'boxed' with the boolean value true, then 'data' is instead the name
+of any non-empty complex type (struct or union), and a pointer to that
+QAPI type is passed as a single argument.
 
 
 === Features ===
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 9aefcfe015..54d02458b5 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -840,7 +840,7 @@ def check_command(expr, info):
 
 args_meta = ['struct']
 if boxed:
-args_meta += ['union', 'alternate']
+args_meta += ['union']
 check_type(info, "'data' for command '%s'" % name,
expr.get('data'), allow_dict=not boxed,
allow_metas=args_meta)
@@ -858,7 +858,7 @@ def check_event(expr, info):
 
 meta = ['struct']
 if boxed:
-meta += ['union', 'alternate']
+meta += ['union']
 check_type(info, "'data' for event '%s'" % name,
expr.get('data'), allow_dict=not boxed,
allow_metas=meta)
@@ -1690,9 +1690,6 @@ class QAPISchemaAlternateType(QAPISchemaType):
 visitor.visit_alternate_type(self.name, self.info, self.ifcond,
  self.variants)
 
-def is_empty(self):
-return False
-
 
 class QAPISchemaCommand(QAPISchemaEntity):
 def __init__(self, name, info, doc, ifcond, arg_type, ret_type,
@@ -1714,15 +1711,13 @@ class QAPISchemaCommand(QAPISchemaEntity):
 QAPISchemaEntity.check(self, schema)
 if self._arg_type_name:
 self.arg_type = schema.lookup_type(self._arg_type_name)
-assert (isinstance(self.arg_type, QAPISchemaObjectType) or
-isinstance(self.arg_type, QAPISchemaAlternateType))
+assert isinstance(self.arg_type, QAPISchemaObjectType)
 self.arg_type.check(schema)
 if self.boxed:
 if self.arg_type.is_empty():
 raise QAPISemError(self.info,
"Cannot use 'boxed' with empty type")
 else:
-assert not isinstance(self.arg_type, QAPISchemaAlternateType)
 assert not self.arg_type.variants
 elif self.boxed:
 raise QAPISemError(self.info, "Use of 'boxed' requires 'data'")
@@ -1750,15 +1745,13 @@ class QAPISchemaEvent(QAPISchemaEntity):
 QAPISchemaEntity.check(self, schema)
 if self._arg_type_name:
 self.arg_type = schema.lookup_type(self._arg_type_name)
-assert (isinstance(self.arg_type, QAPISchemaObjectType) or
- 

[Qemu-devel] [PATCH v3 14/16] docs/devel/qapi-code-gen: Rewrite introduction to schema

[Markus Armbruster]

The introduction to the QAPI schema is somewhat rambling.  Rewrite for
clarity.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt | 107 ---
 1 file changed, 48 insertions(+), 59 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 2406e1bdbc..e75d680f6e 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -16,65 +16,54 @@ well as the QEMU Guest Agent (QGA) for communicating with 
the guest.
 The remainder of this document uses "Client JSON Protocol" when
 referring to the wire contents of a QMP or QGA connection.
 
-To map Client JSON Protocol interfaces to the native C QAPI
-implementations, a JSON-based schema is used to define types and
-function signatures, and a set of scripts is used to generate types,
-signatures, and marshaling/dispatch code. This document will describe
-how the schemas, scripts, and resulting code are used.
-
-
-== QMP/Guest agent schema ==
-
-A QAPI schema file is designed to be loosely based on JSON
-(http://www.ietf.org/rfc/rfc8259.txt) with changes for quoting style
-and the use of comments; a QAPI schema file is then parsed by a python
-code generation program.  A valid QAPI schema consists of a series of
-top-level expressions, with no commas between them.  Where
-dictionaries (JSON objects) are used, they are parsed as python
-OrderedDicts so that ordering is preserved (for predictable layout of
-generated C structs and parameter lists).  Ordering doesn't matter
-between top-level expressions or the keys within an expression, but
-does matter within dictionary values for 'data' and 'returns' members
-of a single expression.  QAPI schema input is written using 'single
-quotes' instead of JSON's "double quotes" (in contrast, Client JSON
-Protocol uses no comments, and while input accepts 'single quotes' as
-an extension, output is strict JSON using only "double quotes").  As
-in JSON, trailing commas are not permitted in arrays or dictionaries.
-Input must be ASCII (although QMP supports full Unicode strings, the
-QAPI parser does not).  At present, there is no place where a QAPI
-schema requires the use of JSON numbers or null.
-
-
-=== Comments ===
-
-Comments are allowed; anything between an unquoted # and the following
-newline is ignored.
-
-
-=== Schema overview ===
-
-The schema sets up a series of types, as well as commands and events
-that will use those types.  Forward references are allowed: the parser
-scans in two passes, where the first pass learns all type names, and
-the second validates the schema and generates the code.  This allows
-the definition of complex structs that can have mutually recursive
-types, and allows for indefinite nesting of Client JSON Protocol that
-satisfies the schema.  A type name should not be defined more than
-once.  It is permissible for the schema to contain additional types
-not used by any commands or events in the Client JSON Protocol, for
-the side effect of generated C code used internally.
-
-There are eight top-level expressions recognized by the parser:
-'include', 'pragma', 'command', 'struct', 'enum', 'union',
-'alternate', and 'event'.  There are several groups of types: simple
-types (a number of built-in types, such as 'int' and 'str'; as well as
-enumerations), complex types (structs and two flavors of unions), and
-alternate types (a choice between other types).  The 'command' and
-'event' expressions can refer to existing types by name, or list an
-anonymous type as a dictionary. Listing a type name inside an array
-refers to a single-dimension array of that type; multi-dimension
-arrays are not directly supported (although an array of a complex
-struct that contains an array member is possible).
+To map between Client JSON Protocol interfaces and the native C API,
+we generate C code from a QAPI schema.  This document describes the
+QAPI schema language, and how it gets mapped to the Client JSON
+Protocol and to C.  It additionally provides guidance on maintaining
+Client JSON Protocol compatibility.
+
+
+== The QAPI schema language ==
+
+The QAPI schema defines the Client JSON Protocol's commands and
+events, as well as types used by them.  Forward references are
+allowed.
+
+It is permissible for the schema to contain additional types not used
+by any commands or events, for the side effect of generated C code
+used internally.
+
+There are several kinds of types: simple types (a number of built-in
+types, such as 'int' and 'str'; as well as enumerations), arrays,
+complex types (structs and two flavors of unions), and alternate types
+(a choice between other types).
+
+
+=== Schema syntax ===
+
+Syntax is loosely based on JSON (http://www.ietf.org/rfc/rfc8259.txt).
+Differences:
+
+* Comments: start with a hash character (#) that is not part of a
+  string, and extend to the end of the line.
+
+* Strings are enclosed in 'single quotes', not "double quotes".
+
+* Strings

[Qemu-devel] [PATCH v3 13/16] docs/devel/qapi-code-gen: Rewrite compatibility considerations

[Markus Armbruster]

We have some compatibility advice buried in sections "Enumeration
types" and "Struct types".  Compatibility is actually about commands
and events.  It devolves to the types used there.  All kinds of types,
not just enumerations and structs.

Replace the existing advice by a new section "Compatibility
considerations".

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt | 95 +++-
 1 file changed, 60 insertions(+), 35 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 663ef10a56..2406e1bdbc 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -178,14 +178,11 @@ While the C code starts numbering at 0, it is better to 
use explicit
 comparisons to enum values than implicit comparisons to 0; the C code
 will also include a generated enum member ending in _MAX for tracking
 the size of the enum, useful when using common functions for
-converting between strings and enum values.  Since the wire format
-always passes by name, it is acceptable to reorder or add new
-enumeration members in any location without breaking clients of Client
-JSON Protocol; however, removing enum values would break
-compatibility.  For any struct that has a member that will only contain
-a finite set of string values, using an enum type for that member is
-better than open-coding the member to be type 'str'.
+converting between strings and enum values.
 
+For any struct that has a member that will only contain a finite set
+of string values, using an enum type for that member is better than
+open-coding the member to be type 'str'.
 
 === Struct types ===
 
@@ -203,34 +200,6 @@ name.  An example of a struct is:
 The use of '*' as a prefix to the name means the member is optional in
 the corresponding JSON protocol usage.
 
-The default initialization value of an optional argument should not be changed
-between versions of QEMU unless the new default maintains backward
-compatibility to the user-visible behavior of the old default.
-
-With proper documentation, this policy still allows some flexibility; for
-example, documenting that a default of 0 picks an optimal buffer size allows
-one release to declare the optimal size at 512 while another release declares
-the optimal size at 4096 - the user-visible behavior is not the bytes used by
-the buffer, but the fact that the buffer was optimal size.
-
-On input structures (only mentioned in the 'data' side of a command), changing
-from mandatory to optional is safe (older clients will supply the option, and
-newer clients can benefit from the default); changing from optional to
-mandatory is backwards incompatible (older clients may be omitting the option,
-and must continue to work).
-
-On output structures (only mentioned in the 'returns' side of a command),
-changing from mandatory to optional is in general unsafe (older clients may be
-expecting the member, and could crash if it is missing), although it
-can be done if the only way that the optional argument will be omitted
-is when it is triggered by the presence of a new input flag to the
-command that older clients don't know to send.  Changing from optional
-to mandatory is safe.
-
-A structure that is used in both input and output of various commands
-must consider the backwards compatibility constraints of both directions
-of use.
-
 A struct definition can specify another struct as its base.
 In this case, the members of the base type are included as top-level members
 of the new struct's dictionary in the Client JSON Protocol wire
@@ -1037,6 +1006,62 @@ the names of built-in types.  Clients should examine 
member
 "json-type" instead of hard-coding names of built-in types.
 
 
+== Compatibility considerations ==
+
+Maintaining backward compatibility at the Client JSON Protocol level
+while evolving the schema requires some care.  This section is about
+syntactic compatibility, which is necessary, but not sufficient, for
+actual compatibility.
+
+Clients send commands with argument data, and receive command
+responses with return data and events with event data.
+
+Adding opt-in functionality to the send direction is backwards
+compatible: adding commands, optional arguments, enumeration values,
+union and alternate branches; turning an argument type into an
+alternate of that type; making mandatory arguments optional.  Clients
+oblivious of the new functionality continue to work.
+
+Incompatible changes include removing commands, command arguments,
+enumeration values, union and alternate branches, adding mandatory
+command arguments, and making optional arguments mandatory.
+
+The specified behavior of an absent optional argument should remain
+the same.  With proper documentation, this policy still allows some
+flexibility; for example, when an optional 'buffer-size' argument is
+specified to default to a sensible buffer size, the actual default
+value can still be changed.  The specifie

[Qemu-devel] [PATCH v3 11/16] qapi: Adjust frontend errors to say enum value, not member

[Markus Armbruster]

For consistency with docs/devel/qapi-code-gen.txt.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py  | 11 ---
 scripts/qapi/events.py  |  2 +-
 tests/qapi-schema/enum-clash-member.err |  2 +-
 tests/qapi-schema/enum-member-case.err  |  2 +-
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 3393a049cc..a538d2f37c 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -1340,7 +1340,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 def __init__(self, name, info, doc, ifcond, members, prefix):
 QAPISchemaType.__init__(self, name, info, doc, ifcond)
 for m in members:
-assert isinstance(m, QAPISchemaMember)
+assert isinstance(m, QAPISchemaEnumMember)
 m.set_owner(name)
 assert prefix is None or isinstance(prefix, str)
 self.members = members
@@ -1551,6 +1551,10 @@ class QAPISchemaMember(object):
 return "'%s' %s" % (self.name, self._pretty_owner())
 
 
+class QAPISchemaEnumMember(QAPISchemaMember):
+role = 'value'
+
+
 class QAPISchemaFeature(QAPISchemaMember):
 role = 'feature'
 
@@ -1807,7 +1811,8 @@ class QAPISchema(object):
 return [QAPISchemaFeature(f['name'], f.get('if')) for f in features]
 
 def _make_enum_members(self, values):
-return [QAPISchemaMember(v['name'], v.get('if')) for v in values]
+return [QAPISchemaEnumMember(v['name'], v.get('if'))
+for v in values]
 
 def _make_implicit_enum_type(self, name, info, ifcond, values):
 # See also QAPISchemaObjectTypeMember._pretty_owner()
@@ -2223,7 +2228,7 @@ const QEnumLookup %(c_name)s_lookup = {
 
 def gen_enum(name, members, prefix=None):
 # append automatically generated _MAX value
-enum_members = members + [QAPISchemaMember('_MAX')]
+enum_members = members + [QAPISchemaEnumMember('_MAX')]
 
 ret = mcgen('''
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index e0abfef7b0..7062553cf3 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -194,7 +194,7 @@ void %(event_emit)s(%(event_enum)s event, QDict *qdict);
   self._event_emit_name))
 # Note: we generate the enum member regardless of @ifcond, to
 # keep the enumeration usable in target-independent code.
-self._event_enum_members.append(QAPISchemaMember(name))
+self._event_enum_members.append(QAPISchemaEnumMember(name))
 
 
 def gen_events(schema, output_dir, prefix):
diff --git a/tests/qapi-schema/enum-clash-member.err 
b/tests/qapi-schema/enum-clash-member.err
index 5403c78507..8238d2e807 100644
--- a/tests/qapi-schema/enum-clash-member.err
+++ b/tests/qapi-schema/enum-clash-member.err
@@ -1 +1 @@
-tests/qapi-schema/enum-clash-member.json:2: 'one_two' (member of MyEnum) 
collides with 'one-two' (member of MyEnum)
+tests/qapi-schema/enum-clash-member.json:2: 'one_two' (value of MyEnum) 
collides with 'one-two' (value of MyEnum)
diff --git a/tests/qapi-schema/enum-member-case.err 
b/tests/qapi-schema/enum-member-case.err
index 3c67a3a067..5d689e92d5 100644
--- a/tests/qapi-schema/enum-member-case.err
+++ b/tests/qapi-schema/enum-member-case.err
@@ -1 +1 @@
-tests/qapi-schema/enum-member-case.json:4: 'Value' (member of 
NoWayThisWillGetWhitelisted) should not use uppercase
+tests/qapi-schema/enum-member-case.json:4: 'Value' (value of 
NoWayThisWillGetWhitelisted) should not use uppercase
-- 
2.21.0

[Qemu-devel] [PATCH v3 00/16] qapi: Schema language cleanups & doc improvements

[Markus Armbruster]

v3:
* PATCH 05
  - Typo fixed [Eric]
* PATCH 06+07
  - Additional comments [Eric]
* PATCH 11
  - Replace one more QAPISchemaMember by QAPISchemaEnumMember
* PATCH 13+15
  - Doc phrasing tweaks [Eric]
* PATCH 14+15
  - Belatedly update for v2's restriction to printable ASCII [Eric]
  - Correct claim "order of top-level expression doesn't matter" [Eric]
* PATCH 15
  - Fix EBNF for PRAGMA [Eric]

v2:
* PATCH v1 05: Dropped
* PATCH 01,05,07-11,16: New
* PATCH 03:
  - Commit message typo fixed
  - Assertions tightened
* PATCH 04:
  - Document member name 'u' is reserved
  - Drop an accidental change to section "Enumeration types"
  - Typo fixed
* PATCH 06:
  - Outlaw anything but printable ASCII, not just control characters
* PATCH 15:
  - Adopt more convenient terminology: directive, definition
  - Explanation of grammar notation missed grouping
  - Drop a more text redundant with the grammar
  - Drop more text redundant with section "Naming rules and reserved
names"
  - Turn "boxed needs data: STRING" into syntax
  - Improve section "Documentation comments" some more
  - Minor phrasing tweaks

Markus Armbruster (16):
  scripts/git.orderfile: Match QAPI schema more precisely
  qapi: Drop check_type()'s redundant parameter @allow_optional
  qapi: Drop support for boxed alternate arguments
  docs/devel/qapi-code-gen: Minor specification fixes
  tests/qapi-schema: Demonstrate bad reporting of funny characters
  qapi: Restrict strings to printable ASCII
  qapi: Drop support for escape sequences other than \\
  qapi: Permit 'boxed' with empty type
  qapi: Permit alternates with just one branch
  qapi: Permit omitting all flat union branches
  qapi: Adjust frontend errors to say enum value, not member
  docs/devel/qapi-code-gen: Reorder sections for readability
  docs/devel/qapi-code-gen: Rewrite compatibility considerations
  docs/devel/qapi-code-gen: Rewrite introduction to schema
  docs/devel/qapi-code-gen: Improve QAPI schema language doc
  qapi: Tweak code to match docs/devel/qapi-code-gen.txt

 docs/devel/qapi-code-gen.txt  | 1071 ++---
 tests/test-qmp-cmds.c |4 +
 scripts/git.orderfile |4 +-
 scripts/qapi/commands.py  |4 +-
 scripts/qapi/common.py|  147 +--
 scripts/qapi/events.py|   12 +-
 tests/Makefile.include|7 +-
 tests/qapi-schema/alternate-empty.err |2 +-
 tests/qapi-schema/alternate-empty.json|4 +-
 tests/qapi-schema/args-boxed-empty.err|1 -
 tests/qapi-schema/args-boxed-empty.json   |3 -
 tests/qapi-schema/args-invalid.err|2 +-
 tests/qapi-schema/doc-missing.err |2 +-
 tests/qapi-schema/doc-no-symbol.err   |2 +-
 tests/qapi-schema/enum-bad-name.err   |2 +-
 tests/qapi-schema/enum-bad-name.json  |3 +-
 tests/qapi-schema/enum-clash-member.err   |2 +-
 .../qapi-schema/enum-dict-member-unknown.err  |2 +-
 tests/qapi-schema/enum-member-case.err|2 +-
 tests/qapi-schema/escape-outside-string.err   |1 -
 tests/qapi-schema/escape-outside-string.json  |3 -
 tests/qapi-schema/escape-too-big.err  |1 -
 tests/qapi-schema/escape-too-big.exit |1 -
 tests/qapi-schema/escape-too-big.json |3 -
 tests/qapi-schema/escape-too-big.out  |0
 tests/qapi-schema/escape-too-short.err|1 -
 tests/qapi-schema/escape-too-short.exit   |1 -
 tests/qapi-schema/escape-too-short.json   |3 -
 tests/qapi-schema/escape-too-short.out|0
 tests/qapi-schema/flat-union-empty.err|2 +-
 tests/qapi-schema/flat-union-empty.json   |2 +-
 tests/qapi-schema/ident-with-escape.err   |1 +
 tests/qapi-schema/ident-with-escape.exit  |2 +-
 tests/qapi-schema/ident-with-escape.json  |2 +-
 tests/qapi-schema/ident-with-escape.out   |   16 -
 tests/qapi-schema/pragma-non-dict.err |2 +-
 tests/qapi-schema/qapi-schema-test.json   |   13 +-
 tests/qapi-schema/qapi-schema-test.out|   21 +-
 tests/qapi-schema/string-code-point-127.err   |1 +
 ...-empty.exit => string-code-point-127.exit} |0
 tests/qapi-schema/string-code-point-127.json  |2 +
 ...ed-empty.out => string-code-point-127.out} |0
 tests/qapi-schema/string-code-point-31.err|1 +
 ...-string.exit => string-code-point-31.exit} |0
 tests/qapi-schema/string-code-point-31.json   |2 +
 ...de-string.out => string-code-point-31.out} |0
 tests/qapi-schema/struct-data-invalid.err |2 +-
 tests/qapi-schema/unicode-str.err |1 -
 tests/qapi-schema/unicode-str.exit|1 -
 tests/qapi-schema/unicode-str.json|2 -
 tests/qapi-schema/unicode-str.out |0
 tests/qapi-schema/union-empty.err |2 +-
 tests/qapi-schema/union-empty.

[Qemu-devel] [PATCH v3 01/16] scripts/git.orderfile: Match QAPI schema more precisely

[Markus Armbruster]

Pattern *.json also matches the tests/qapi-schema/*.json.  Separates
them from the tests/qapi-schema/*.{err,exit,out} in diffs.  I hate
that.  Change the pattern to match just the "real" QAPI schemata.

Signed-off-by: Markus Armbruster 
Reviewed-by: Philippe Mathieu-Daudé 
---
 scripts/git.orderfile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/git.orderfile b/scripts/git.orderfile
index ac699700b1..e89790941c 100644
--- a/scripts/git.orderfile
+++ b/scripts/git.orderfile
@@ -19,11 +19,11 @@ Makefile*
 *.mak
 
 # qapi schema
-*.json
+qapi/*.json
+qga/*.json
 
 # headers
 *.h
 
 # code
 *.c
-
-- 
2.21.0

[Qemu-devel] [PATCH v3 07/16] qapi: Drop support for escape sequences other than \\

[Markus Armbruster]

Since the previous commit restricted strings to printable ASCII,
\u's only use is obfuscation.  Drop it.

This leaves \\, \/, \', and \".  Since QAPI schema strings are all
names, and names are restricted to ASCII letters, digits, hyphen, and
underscore, none of them is useful.

The latter three have no test coverage.  Drop them.

Keep \\ to avoid (more) gratuitous incompatibility with JSON.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py   | 26 +++-
 tests/Makefile.include   |  3 ---
 tests/qapi-schema/escape-outside-string.err  |  1 -
 tests/qapi-schema/escape-outside-string.exit |  1 -
 tests/qapi-schema/escape-outside-string.json |  3 ---
 tests/qapi-schema/escape-outside-string.out  |  0
 tests/qapi-schema/escape-too-big.err |  1 -
 tests/qapi-schema/escape-too-big.exit|  1 -
 tests/qapi-schema/escape-too-big.json|  3 ---
 tests/qapi-schema/escape-too-big.out |  0
 tests/qapi-schema/escape-too-short.err   |  1 -
 tests/qapi-schema/escape-too-short.exit  |  1 -
 tests/qapi-schema/escape-too-short.json  |  3 ---
 tests/qapi-schema/escape-too-short.out   |  0
 tests/qapi-schema/ident-with-escape.err  |  1 +
 tests/qapi-schema/ident-with-escape.exit |  2 +-
 tests/qapi-schema/ident-with-escape.json |  2 +-
 tests/qapi-schema/ident-with-escape.out  | 16 
 tests/qapi-schema/unknown-escape.json|  2 +-
 19 files changed, 7 insertions(+), 60 deletions(-)
 delete mode 100644 tests/qapi-schema/escape-outside-string.err
 delete mode 100644 tests/qapi-schema/escape-outside-string.exit
 delete mode 100644 tests/qapi-schema/escape-outside-string.json
 delete mode 100644 tests/qapi-schema/escape-outside-string.out
 delete mode 100644 tests/qapi-schema/escape-too-big.err
 delete mode 100644 tests/qapi-schema/escape-too-big.exit
 delete mode 100644 tests/qapi-schema/escape-too-big.json
 delete mode 100644 tests/qapi-schema/escape-too-big.out
 delete mode 100644 tests/qapi-schema/escape-too-short.err
 delete mode 100644 tests/qapi-schema/escape-too-short.exit
 delete mode 100644 tests/qapi-schema/escape-too-short.json
 delete mode 100644 tests/qapi-schema/escape-too-short.out

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 539b50f9ac..0fb1d1956a 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -524,29 +524,9 @@ class QAPISchemaParser(object):
 if ch == '\n':
 raise QAPIParseError(self, 'Missing terminating "\'"')
 if esc:
-# Note: we don't recognize escape sequences
-# for control characters
-if ch == 'u':
-value = 0
-for _ in range(0, 4):
-ch = self.src[self.cursor]
-self.cursor += 1
-if ch not in '0123456789abcdefABCDEF':
-raise QAPIParseError(self,
- '\\u escape needs 4 '
- 'hex digits')
-value = (value << 4) + int(ch, 16)
-# If Python 2 and 3 didn't disagree so much on
-# how to handle Unicode, then we could allow
-# Unicode string defaults.  But most of QAPI is
-# ASCII-only, so we aren't losing much for now.
-if not value or value > 0x7f:
-raise QAPIParseError(self,
- 'For now, \\u escape '
- 'only supports non-zero '
- 'values up to \\u007f')
-ch = chr(value)
-elif ch not in '\\/\'"':
+# Note: we recognize only \\ because we have
+# no use for funny characters in strings
+if ch != '\\':
 raise QAPIParseError(self,
  "Unknown escape \\%s" % ch)
 esc = False
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 17c53bed52..023a4e4789 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -375,9 +375,6 @@ qapi-schema += enum-int-member.json
 qapi-schema += enum-member-case.json
 qapi-schema += enum-missing-data.json
 qapi-schema += enum-wrong-data.json
-qapi-schema += escape-outside-string.json
-qapi-schema += escape-too-big.json
-qapi-schema += escape-too-short.json
 qapi-schema += event-boxed-empty.json
 qapi-schema += event-case.json
 qapi-schema += event-member-invali

[Qemu-devel] [PATCH v3 06/16] qapi: Restrict strings to printable ASCII

[Markus Armbruster]

RFC 8259 on string contents:

   All Unicode characters may be placed within the quotation marks,
   except for the characters that MUST be escaped: quotation mark,
   reverse solidus, and the control characters (U+ through
   U+001F).

The QAPI schema parser accepts both less and more than JSON: it
accepts only ASCII with \u (less), and accepts control characters
other than LF (new line) unescaped.  How it treats unescaped non-ASCII
input differs between Python 2 and Python 3.

Make it accept strictly less: require printable ASCII.  Drop support
for \b, \f, \n, \r, \t.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py| 28 ---
 tests/Makefile.include|  3 +-
 tests/qapi-schema/string-code-point-127.err   |  1 +
 ...de-str.exit => string-code-point-127.exit} |  0
 tests/qapi-schema/string-code-point-127.json  |  2 ++
 ...code-str.out => string-code-point-127.out} |  0
 tests/qapi-schema/string-code-point-31.err|  1 +
 tests/qapi-schema/string-code-point-31.exit   |  1 +
 tests/qapi-schema/string-code-point-31.json   |  2 ++
 tests/qapi-schema/string-code-point-31.out|  0
 tests/qapi-schema/unicode-str.err |  1 -
 tests/qapi-schema/unicode-str.json|  2 --
 12 files changed, 20 insertions(+), 21 deletions(-)
 create mode 100644 tests/qapi-schema/string-code-point-127.err
 rename tests/qapi-schema/{unicode-str.exit => string-code-point-127.exit} 
(100%)
 create mode 100644 tests/qapi-schema/string-code-point-127.json
 rename tests/qapi-schema/{unicode-str.out => string-code-point-127.out} (100%)
 create mode 100644 tests/qapi-schema/string-code-point-31.err
 create mode 100644 tests/qapi-schema/string-code-point-31.exit
 create mode 100644 tests/qapi-schema/string-code-point-31.json
 create mode 100644 tests/qapi-schema/string-code-point-31.out
 delete mode 100644 tests/qapi-schema/unicode-str.err
 delete mode 100644 tests/qapi-schema/unicode-str.json

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 54d02458b5..539b50f9ac 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -515,6 +515,7 @@ class QAPISchemaParser(object):
 elif self.tok in '{}:,[]':
 return
 elif self.tok == "'":
+# Note: we accept only printable ASCII
 string = ''
 esc = False
 while True:
@@ -523,17 +524,9 @@ class QAPISchemaParser(object):
 if ch == '\n':
 raise QAPIParseError(self, 'Missing terminating "\'"')
 if esc:
-if ch == 'b':
-string += '\b'
-elif ch == 'f':
-string += '\f'
-elif ch == 'n':
-string += '\n'
-elif ch == 'r':
-string += '\r'
-elif ch == 't':
-string += '\t'
-elif ch == 'u':
+# Note: we don't recognize escape sequences
+# for control characters
+if ch == 'u':
 value = 0
 for _ in range(0, 4):
 ch = self.src[self.cursor]
@@ -552,20 +545,21 @@ class QAPISchemaParser(object):
  'For now, \\u escape '
  'only supports non-zero '
  'values up to \\u007f')
-string += chr(value)
-elif ch in '\\/\'"':
-string += ch
-else:
+ch = chr(value)
+elif ch not in '\\/\'"':
 raise QAPIParseError(self,
  "Unknown escape \\%s" % ch)
 esc = False
 elif ch == '\\':
 esc = True
+continue
 elif ch == "'":
 self.val = string
 return
-else:
-string += ch
+if ord(ch) < 32 or ord(ch) >= 127:
+raise QAPIParseError(
+self, "Funny character in string")
+string += ch
 elif self.src.startswith('true', self.pos):
 self.val = True
 self.cursor += 3
diff --git a/tests/Makefile.include b/tests/Makefile.include
index b39860a8d0..17c53bed52 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -452,6 +452,8 @@ qapi-schema += returns-array-bad.json
 qapi-schema +=

[Qemu-devel] [PATCH v3 02/16] qapi: Drop check_type()'s redundant parameter @allow_optional

[Markus Armbruster]

check_type() uses @allow_optional only when @value is a dictionary and
@allow_dict is True.  All callers that pass allow_dict=True also pass
allow_optional=True.

Therefore, @allow_optional is always True when check_type() uses it.
Drop the redundant parameter.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 scripts/qapi/common.py | 17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index d61bfdc526..9aefcfe015 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -783,9 +783,8 @@ def check_if(expr, info):
 check_if_str(ifcond, info)
 
 
-def check_type(info, source, value, allow_array=False,
-   allow_dict=False, allow_optional=False,
-   allow_metas=[]):
+def check_type(info, source, value,
+   allow_array=False, allow_dict=False, allow_metas=[]):
 global all_names
 
 if value is None:
@@ -821,7 +820,7 @@ def check_type(info, source, value, allow_array=False,
 # value is a dictionary, check that each member is okay
 for (key, arg) in value.items():
 check_name(info, "Member of %s" % source, key,
-   allow_optional=allow_optional)
+   allow_optional=True)
 if c_name(key, False) == 'u' or c_name(key, False).startswith('has_'):
 raise QAPISemError(info, "Member of %s uses reserved name '%s'"
% (source, key))
@@ -843,14 +842,14 @@ def check_command(expr, info):
 if boxed:
 args_meta += ['union', 'alternate']
 check_type(info, "'data' for command '%s'" % name,
-   expr.get('data'), allow_dict=not boxed, allow_optional=True,
+   expr.get('data'), allow_dict=not boxed,
allow_metas=args_meta)
 returns_meta = ['union', 'struct']
 if name in returns_whitelist:
 returns_meta += ['built-in', 'alternate', 'enum']
 check_type(info, "'returns' for command '%s'" % name,
expr.get('returns'), allow_array=True,
-   allow_optional=True, allow_metas=returns_meta)
+   allow_metas=returns_meta)
 
 
 def check_event(expr, info):
@@ -861,7 +860,7 @@ def check_event(expr, info):
 if boxed:
 meta += ['union', 'alternate']
 check_type(info, "'data' for event '%s'" % name,
-   expr.get('data'), allow_dict=not boxed, allow_optional=True,
+   expr.get('data'), allow_dict=not boxed,
allow_metas=meta)
 
 
@@ -889,7 +888,7 @@ def check_union(expr, info):
 else:
 # The object must have a string or dictionary 'base'.
 check_type(info, "'base' for union '%s'" % name,
-   base, allow_dict=True, allow_optional=True,
+   base, allow_dict=True,
allow_metas=['struct'])
 if not base:
 raise QAPISemError(info, "Flat union '%s' must have a base"
@@ -1012,7 +1011,7 @@ def check_struct(expr, info):
 features = expr.get('features')
 
 check_type(info, "'data' for struct '%s'" % name, members,
-   allow_dict=True, allow_optional=True)
+   allow_dict=True)
 check_type(info, "'base' for struct '%s'" % name, expr.get('base'),
allow_metas=['struct'])
 
-- 
2.21.0

[Qemu-devel] [PATCH v3 04/16] docs/devel/qapi-code-gen: Minor specification fixes

[Markus Armbruster]

The specification claims "Each expression that isn't an include
directive may be preceded by a documentation block", but the code also
rejects them for pragma directives.  The code is correct.  Fix the
specification.

The specification reserves member names starting with 'has_', but the
code also reserves name 'u'.  Fix the specification.

The specification claims "The string 'max' is not allowed as an enum
value".  Untrue.  Fix the specification.  While there, delete the
naming advice, because it's redundant with the naming rules in section
"Schema overview"

The specification claims "No branch of the union can be named 'max',
as this would collide with the implicit enum".  Untrue.  Fix the
specification.

The specification claims "It is not allowed to name an event 'MAX',
since the generator also produces a C enumeration of all event names
with a generated _MAX value at the end."  Untrue.  Fix the
specification.

The specification claims "All branches of the union must be complex
types", but the code permits only struct types.  The code is correct.
Fix the specification.

The specification claims a command's return type "must be the string
name of a complex or built-in type, a one-element array containing the
name of a complex or built-in type" unless the command is in pragma
'returns-whitelist'.  The code does not permit built-in types.  Fix
the specification.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 docs/devel/qapi-code-gen.txt | 34 --
 1 file changed, 16 insertions(+), 18 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 3d3931fb7a..4ce67752a7 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -117,9 +117,9 @@ Example:
 
  Expression documentation 
 
-Each expression that isn't an include directive may be preceded by a
-documentation block.  Such blocks are called expression documentation
-blocks.
+Expressions other than include and pragma directives may be preceded
+by a documentation block.  Such blocks are called expression
+documentation blocks.
 
 When documentation is required (see pragma 'doc-required'), expression
 documentation blocks are mandatory.
@@ -243,8 +243,9 @@ underscore.
 
 Event names should be ALL_CAPS with words separated by underscore.
 
-Member names starting with 'has-' or 'has_' are reserved for the
-generator, which uses them for tracking optional members.
+Member name 'u' and names starting with 'has-' or 'has_' are reserved
+for the generator, which uses them for unions and for tracking
+optional members.
 
 Any name (command, event, type, member, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
@@ -460,15 +461,14 @@ discriminator value, as in these examples:
 
 The generated C code uses a struct containing a union. Additionally,
 an implicit C enum 'NameKind' is created, corresponding to the union
-'Name', for accessing the various branches of the union.  No branch of
-the union can be named 'max', as this would collide with the implicit
-enum.  The value for each branch can be of any type.
+'Name', for accessing the various branches of the union.  The value
+for each branch can be of any type.
 
 A flat union definition avoids nesting on the wire, and specifies a
 set of common members that occur in all variants of the union.  The
 'base' key must specify either a type name (the type must be a
 struct, not a union), or a dictionary representing an anonymous type.
-All branches of the union must be complex types, and the top-level
+All branches of the union must be struct types, and the top-level
 members of the union dictionary on the wire will be combination of
 members from both the base type and the appropriate branch type (when
 merging two dictionaries, there must be no keys in common).  The
@@ -578,8 +578,8 @@ The 'returns' member describes what will appear in the 
"return" member
 of a Client JSON Protocol reply on successful completion of a command.
 The member is optional from the command declaration; if absent, the
 "return" member will be an empty dictionary.  If 'returns' is present,
-it must be the string name of a complex or built-in type, a
-one-element array containing the name of a complex or built-in type.
+it must be the string name of a complex type, or a
+one-element array containing the name of a complex type.
 To return anything else, you have to list the command in pragma
 'returns-whitelist'.  If you do this, the command cannot be extended
 to return additional information in the future.  Use of
@@ -691,13 +691,11 @@ started with --preconfig.
 Usage: { 'event': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
  '*boxed': true }
 
-Events are defined with the keyword 'event'.  It is not allowed to
-name an event 'MAX', since the generator also produces a C enumeration
-of all event names with a generated _MAX value at the end.  When
-'data' is also specified, additiona

[Qemu-devel] [PATCH v3 05/16] tests/qapi-schema: Demonstrate bad reporting of funny characters

[Markus Armbruster]

Invalid name 'not\\possible' is reported as 'not\possible'.  Control
characters (quoted or not) are even more confusing.  Mark FIXME.

Signed-off-by: Markus Armbruster 
Reviewed-by: Eric Blake 
---
 tests/qapi-schema/enum-bad-name.err  | 2 +-
 tests/qapi-schema/enum-bad-name.json | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qapi-schema/enum-bad-name.err 
b/tests/qapi-schema/enum-bad-name.err
index 9c3c1002b7..26a09f84ad 100644
--- a/tests/qapi-schema/enum-bad-name.err
+++ b/tests/qapi-schema/enum-bad-name.err
@@ -1 +1 @@
-tests/qapi-schema/enum-bad-name.json:2: Member of enum 'MyEnum' uses invalid 
name 'not^possible'
+tests/qapi-schema/enum-bad-name.json:3: Member of enum 'MyEnum' uses invalid 
name 'not\possible'
diff --git a/tests/qapi-schema/enum-bad-name.json 
b/tests/qapi-schema/enum-bad-name.json
index 8506562b31..1c4620edda 100644
--- a/tests/qapi-schema/enum-bad-name.json
+++ b/tests/qapi-schema/enum-bad-name.json
@@ -1,2 +1,3 @@
 # we ensure all enum names can map to C
-{ 'enum': 'MyEnum', 'data': [ 'not^possible' ] }
+# FIXME reports 'not\possible' instead of 'not\\possible'
+{ 'enum': 'MyEnum', 'data': [ 'not\\possible' ] }
-- 
2.21.0

Re: [Qemu-devel] [Qemu-block] [PATCH 0/2] trace: Forbid trailing newline in event format

[John Snow]

On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote:
> Hi Stefan,
> 
> I'v been confused by trailing newline in trace reports,
> so this series aims to fix this, by cleaning current
> formats and add a check to catch new one introduced.
> 
> Regards,
> 
> Phil.
> 
> Philippe Mathieu-Daudé (2):
>   trace: Remove trailing newline in events
>   trace: Forbid event format ending with newline character
> 
>  docs/devel/tracing.txt|  2 ++
>  hw/misc/trace-events  | 10 +-
>  hw/scsi/trace-events  |  2 +-
>  hw/sd/trace-events|  2 +-
>  nbd/trace-events  |  4 ++--
>  net/trace-events  |  6 +++---
>  scripts/tracetool/__init__.py |  3 +++
>  7 files changed, 17 insertions(+), 12 deletions(-)
> 

Never mind my bikeshedding.

Reviewed-by: John Snow 

Re: [Qemu-devel] [PATCH 15/15] pc: Add an SMB0 ACPI device to q35

[Corey Minyard]

On Mon, Aug 19, 2019 at 03:17:05PM -0500, miny...@acm.org wrote:
> From: Corey Minyard 
> 
> This is so I2C devices can be found in the ACPI namespace.  Currently
> that's only IPMI, but devices can be easily added now.
> 
> Adding the devices required some PCI information, and the bus itself
> to be added to the PCMachineState structure.
> 
> Note that this only works on Q35, the ACPI for PIIX4 is not capable
> of handling an SMBus device.

I haven't recevied any comments on this, and it seems like something
some people should review.  Is this ok?

Thanks

-corey

> 
> Cc: Michael S. Tsirkin 
> Cc: Igor Mammedov 
> Signed-off-by: Corey Minyard 
> ---
>  hw/i386/acpi-build.c |  15 +++
>  hw/i386/pc_piix.c|  12 ++--
>  hw/i386/pc_q35.c |   9 +
>  include/hw/i386/pc.h |   2 ++
>  tests/data/acpi/q35/DSDT | Bin 7841 -> 7879 bytes
>  tests/data/acpi/q35/DSDT.bridge  | Bin 7858 -> 7896 bytes
>  tests/data/acpi/q35/DSDT.cphp| Bin 8304 -> 8342 bytes
>  tests/data/acpi/q35/DSDT.dimmpxm | Bin 9494 -> 9532 bytes
>  tests/data/acpi/q35/DSDT.ipmibt  | Bin 7916 -> 7954 bytes
>  tests/data/acpi/q35/DSDT.memhp   | Bin 9200 -> 9238 bytes
>  tests/data/acpi/q35/DSDT.mmio64  | Bin 8971 -> 9009 bytes
>  tests/data/acpi/q35/DSDT.numamem | Bin 7847 -> 7885 bytes
>  12 files changed, 28 insertions(+), 10 deletions(-)
> 
> diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
> index 0c94e21a1a..e5fb3d6ef0 100644
> --- a/hw/i386/acpi-build.c
> +++ b/hw/i386/acpi-build.c
> @@ -1809,6 +1809,18 @@ static Aml *build_q35_osc_method(void)
>  return method;
>  }
>  
> +static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)
> +{
> +Aml *scope = aml_scope("_SB.PCI0");
> +Aml *dev = aml_device("SMB0");
> +
> +aml_append(dev, aml_name_decl("_HID", aml_eisaid("APP0005")));
> +aml_append(dev, aml_name_decl("_ADR", aml_int(devnr << 16 | func)));
> +build_acpi_ipmi_devices(dev, BUS(smbus), "\\_SB.PCI0.SMB0");
> +aml_append(scope, dev);
> +aml_append(table, scope);
> +}
> +
>  static void
>  build_dsdt(GArray *table_data, BIOSLinker *linker,
> AcpiPmInfo *pm, AcpiMiscInfo *misc,
> @@ -1862,6 +1874,9 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
>  build_q35_isa_bridge(dsdt);
>  build_isa_devices_aml(dsdt);
>  build_q35_pci0_int(dsdt);
> +if (pcms->smbus && !pcmc->do_not_add_smb_acpi) {
> +build_smb0(dsdt, pcms->smbus, ICH9_SMB_DEV, ICH9_SMB_FUNC);
> +}
>  }
>  
>  if (pcmc->legacy_cpu_hotplug) {
> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> index 9e187f856a..96311b0a91 100644
> --- a/hw/i386/pc_piix.c
> +++ b/hw/i386/pc_piix.c
> @@ -283,15 +283,14 @@ else {
>  
>  if (pcmc->pci_enabled && acpi_enabled) {
>  DeviceState *piix4_pm;
> -I2CBus *smbus;
>  
>  smi_irq = qemu_allocate_irq(pc_acpi_smi_interrupt, first_cpu, 0);
>  /* TODO: Populate SPD eeprom data.  */
> -smbus = piix4_pm_init(pci_bus, piix3_devfn + 3, 0xb100,
> -  pcms->gsi[9], smi_irq,
> -  pc_machine_is_smm_enabled(pcms),
> -  &piix4_pm);
> -smbus_eeprom_init(smbus, 8, NULL, 0);
> +pcms->smbus = piix4_pm_init(pci_bus, piix3_devfn + 3, 0xb100,
> +pcms->gsi[9], smi_irq,
> +pc_machine_is_smm_enabled(pcms),
> +&piix4_pm);
> +smbus_eeprom_init(pcms->smbus, 8, NULL, 0);
>  
>  object_property_add_link(OBJECT(machine), 
> PC_MACHINE_ACPI_DEVICE_PROP,
>   TYPE_HOTPLUG_HANDLER,
> @@ -464,6 +463,7 @@ static void pc_i440fx_3_1_machine_options(MachineClass *m)
>  
>  pc_i440fx_4_0_machine_options(m);
>  m->is_default = 0;
> +pcmc->do_not_add_smb_acpi = true;
>  m->smbus_no_migration_support = true;
>  m->alias = NULL;
>  pcmc->pvh_enabled = false;
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index be3464f485..7ce4fb6fdb 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -316,10 +316,10 @@ static void pc_q35_init(MachineState *machine)
>  
>  if (pcms->smbus_enabled) {
>  /* TODO: Populate SPD eeprom data.  */
> -smbus_eeprom_init(ich9_smb_init(host_bus,
> -PCI_DEVFN(ICH9_SMB_DEV, 
> ICH9_SMB_FUNC),
> -0xb100),
> -  8, NULL, 0);
> +pcms->smbus = ich9_smb_init(host_bus,
> +PCI_DEVFN(ICH9_SMB_DEV, ICH9_SMB_FUNC),
> +0xb100);
> +smbus_eeprom_init(pcms->smbus, 8, NULL, 0);
>  }
>  
>  pc_cmos_init(pcms, idebus[0], idebus[1], rtc_state);
> @@ -410,6 +410,7 @@ static void pc_q35_3_1_machine_options(MachineClass *m)
>  

Re: [Qemu-devel] [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

[John Snow]

On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote:
> Event format ending with newlines confuse the trace reports.
> Forbid them.
> 
> Add a check to refuse new format added with trailing newline:
> 
>   $ make
>   [...]
> GEN hw/misc/trace.h
>   Traceback (most recent call last):
> File "scripts/tracetool.py", line 152, in 
>   main(sys.argv)
> File "scripts/tracetool.py", line 143, in main
>   events.extend(tracetool.read_events(fh, arg))
> File "scripts/tracetool/__init__.py", line 367, in read_events
>   event = Event.build(line)
> File "scripts/tracetool/__init__.py", line 281, in build
>   raise ValueError("Event format can not end with a newline character")
>   ValueError: Error at hw/misc/trace-events:121: Event format can not end 
> with a newline character
> 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  docs/devel/tracing.txt| 2 ++
>  scripts/tracetool/__init__.py | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/docs/devel/tracing.txt b/docs/devel/tracing.txt
> index 76e492a489..8231bbf5d1 100644
> --- a/docs/devel/tracing.txt
> +++ b/docs/devel/tracing.txt
> @@ -112,6 +112,8 @@ Trace events should use types as follows:
>  Format strings should reflect the types defined in the trace event.  Take
>  special care to use PRId64 and PRIu64 for int64_t and uint64_t types,
>  respectively.  This ensures portability between 32- and 64-bit platforms.
> +Format strings must not end with a newline character.  It is the 
> responsibility
> +of backends to adapt line ending for proper logging.
>  
>  Each event declaration will start with the event name, then its arguments,
>  finally a format string for pretty-printing. For example:
> diff --git a/scripts/tracetool/__init__.py b/scripts/tracetool/__init__.py
> index 6fca674936..57df74e67c 100644
> --- a/scripts/tracetool/__init__.py
> +++ b/scripts/tracetool/__init__.py
> @@ -277,6 +277,9 @@ class Event(object):
>  if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
>  raise ValueError("Event format '%m' is forbidden, pass the error 
> "
>   "as an explicit trace argument")
> +if fmt.endswith("\\n\""):
> +raise ValueError("Event format must not end with a newline "

It's barely worth mentioning, but you can use r"\n" for cases like this,
if it makes it easier to read.

Re: [Qemu-devel] [Qemu-block] [PATCH v2 1/2] blockdev: release the AioContext at drive_backup_prepare

[John Snow]

On 9/13/19 11:25 AM, Sergio Lopez wrote:
> do_drive_backup() already acquires the AioContext, so release it
> before the call.
> 
> Signed-off-by: Sergio Lopez 
> ---
>  blockdev.c | 6 +-
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/blockdev.c b/blockdev.c
> index fbef6845c8..3927fdab80 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -1783,20 +1783,16 @@ static void drive_backup_prepare(BlkActionState 
> *common, Error **errp)
>  
>  aio_context = bdrv_get_aio_context(bs);
>  aio_context_acquire(aio_context);
> -
>  /* Paired with .clean() */
>  bdrv_drained_begin(bs);

Do we need to make this change to blockdev_backup_prepare as well?

> +aio_context_release(aio_context);
>  
>  state->bs = bs;
>  
>  state->job = do_drive_backup(backup, common->block_job_txn, &local_err);
>  if (local_err) {
>  error_propagate(errp, local_err);
> -goto out;
>  }
> -
> -out:
> -aio_context_release(aio_context);
>  }
>  
>  static void drive_backup_commit(BlkActionState *common)
> 

[Qemu-devel] [PATCH] podman: fix command invocation

[John Snow]

Oops; there's no argv here.

Signed-off-by: John Snow 
---
 tests/docker/docker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 29613afd48..bc7a470ca2 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -334,7 +334,7 @@ class Docker(object):
 cmd = [ "-u", str(uid) ] + cmd
 # podman requires a bit more fiddling
 if self._command[0] == "podman":
-argv.insert(0, '--userns=keep-id')
+cmd.insert(0, '--userns=keep-id')
 
 ret = self._do_check(["run", "--label",
  "com.qemu.instance.uuid=" + label] + cmd,
-- 
2.21.0

[Qemu-devel] Python3 support for patches

[John Snow]

Hi,

I quite like the patches tool; but python2 is notably doomed.

I tried my hand at polishing it up for python3 and pushed the results
here: https://github.com/jnsnow/patches/tree/python3

I didn't faff around with trying to add simultaneous support: this is a
direct conversion in one big chunk. I used 2to3 and supplemented with
pylint and flake8 to find problem spots.

I don't use all of the features of this tool, but maybe if you'd like to
give this a try you can test this branch and let me know if I missed any
spots.

You can install patches 0.3.1 using the standard incantations:
> python3 setup.py install --user

If it's too broken, you can uninstall it later with:
> pip3 uninstall patches

You may need to check back to the master branch and force a
reinstallation of the python2 version to get the bin back in your PATH:

> git checkout master
> python2 setup.py install --user


Happy python2 doomsday,
--js

Re: [Qemu-devel] [PATCH v2] util/hbitmap: strict hbitmap_reset

[John Snow]

On 9/12/19 4:20 AM, Vladimir Sementsov-Ogievskiy wrote:
> 11.09.2019 20:59, John Snow wrote:
>>
>>
>> On 9/11/19 11:13 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> 07.08.2019 19:27, John Snow wrote:


 On 8/6/19 12:19 PM, Vladimir Sementsov-Ogievskiy wrote:
> 06.08.2019 19:09, Max Reitz wrote:
>> On 06.08.19 17:26, Vladimir Sementsov-Ogievskiy wrote:
>>> hbitmap_reset has an unobvious property: it rounds requested region up.
>>> It may provoke bugs, like in recently fixed write-blocking mode of
>>> mirror: user calls reset on unaligned region, not keeping in mind that
>>> there are possible unrelated dirty bytes, covered by rounded-up region
>>> and information of this unrelated "dirtiness" will be lost.
>>>
>>> Make hbitmap_reset strict: assert that arguments are aligned, allowing
>>> only one exception when @start + @count == hb->orig_size. It's needed
>>> to comfort users of hbitmap_next_dirty_area, which cares about
>>> hb->orig_size.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>>> ---
>>>
>>> v2 based on Max's https://github.com/XanClic/qemu.git block
>>> which will be merged soon to 4.1, and this patch goes to 4.2
>>> Based-on: https://github.com/XanClic/qemu.git block
>>>
>>> v1 was "[PATCH] util/hbitmap: fix unaligned reset", and as I understand
>>> we all agreed to just assert alignment instead of aligning down
>>> automatically.
>>>
>>> include/qemu/hbitmap.h | 5 +
>>> tests/test-hbitmap.c   | 2 +-
>>> util/hbitmap.c | 4 
>>> 3 files changed, 10 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
>>> index 4afbe6292e..7865e819ca 100644
>>> --- a/include/qemu/hbitmap.h
>>> +++ b/include/qemu/hbitmap.h
>>> @@ -132,6 +132,11 @@ void hbitmap_set(HBitmap *hb, uint64_t start, 
>>> uint64_t count);
>>>  * @count: Number of bits to reset.
>>>  *
>>>  * Reset a consecutive range of bits in an HBitmap.
>>> + * @start and @count must be aligned to bitmap granularity. The only 
>>> exception
>>> + * is resetting the tail of the bitmap: @count may be equal to @start +
>>> + * hb->orig_size,
>>
>> s/@start + hb->orig_size/hb->orig_size - @start/, I think.
>
> Ha, I wanted to say start + count equal to orig_size. Yours is OK too of 
> course.
>
>>
>>>in this case @count may be not aligned. @start + @count
>>
>> +are
>>
>> With those fixed:
>>
>> Reviewed-by: Max Reitz 
>
> Thanks!
>

 I'll add this to the pile for 4.2, after I fix the rebase conflicts that
 arose from 4.1-rc4.

>>>
>>> Hi!
>>>
>>> Didn't you forget, or should I resend?
>>>
>>>
>>
>> I must have dropped the patch by accident during the rebasing. As an
>> apology, I squashed in Max's suggestions from the list. Check that they
>> look OK, please?
>>
>> Thanks, applied to my bitmaps tree:
>>
>> https://github.com/jnsnow/qemu/commits/bitmaps
>> https://github.com/jnsnow/qemu.git
>>
> 
> Thanks! Still:
> 
> Quote from your branch:
> 
>  >   * Reset a consecutive range of bits in an HBitmap.
>  > + * @start and @count must be aligned to bitmap granularity. The only 
> exception
>  > + * is resetting the tail of the bitmap: @count may be equal to 
> hb->orig_size -
>  > + * start, in this case @count may be not aligned. @start + @count are
> 
> s/start/@start/ (corresponds to Max's comment, too)
> 

OK, you got it.

> Also, I'm not sure about "are" suggested by Max. "are" is for plural, but 
> here I meant
> one object: sum of @start and @count.
> 

There's not great agreement universally about how to treat things like
collective nouns. Sometimes "Data" is singular, but sometimes it's
plural. "It depends."

In this case, "start + count" refers to one sum, but two constituent
pieces, so it's functioning like a collective noun.

We might say "a + b (together) /are/ ..." but also "the sum of a + b /is/".

> So, you may use exactly "Sum of @start and @count is" or "(@start + @count) 
> sum is" or
> just "(@start + @count) is", whichever you like more.
> 

I like using "the sum of @x and @y is" for being grammatically unambiguous.

updated and pushed.

(Sorry about my language again! --js)

[Qemu-devel] [Bug 1843941] [NEW] RBD Namespaces are not supported

[Gregory O'Neill]

Public bug reported:

Ceph Nautilus (v14.2.0) introduced the Namespaces concept for RADOS
Block Devices. This provides a logical separation within a RADOS Pool
for RBD images which enables granular access control. See
https://docs.ceph.com/docs/nautilus/releases/nautilus/ for additional
details.

librados and librbd support this, however qemu does not. The rbd man
page defines how rbd images within a namespace can be referenced.
https://docs.ceph.com/docs/nautilus/man/8/rbd/#image-snap-group-and-
journal-specs

Adding support for RBD namespaces would be beneficial for security and
reducing the impact of a hypervisor being compromised and putting an
entire Ceph pool or cluster at risk.

** Affects: qemu
 Importance: Undecided
 Status: New


** Tags: rbd

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1843941

Title:
  RBD Namespaces are not supported

Status in QEMU:
  New

Bug description:
  Ceph Nautilus (v14.2.0) introduced the Namespaces concept for RADOS
  Block Devices. This provides a logical separation within a RADOS Pool
  for RBD images which enables granular access control. See
  https://docs.ceph.com/docs/nautilus/releases/nautilus/ for additional
  details.

  librados and librbd support this, however qemu does not. The rbd man
  page defines how rbd images within a namespace can be referenced.
  https://docs.ceph.com/docs/nautilus/man/8/rbd/#image-snap-group-and-
  journal-specs

  Adding support for RBD namespaces would be beneficial for security and
  reducing the impact of a hypervisor being compromised and putting an
  entire Ceph pool or cluster at risk.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1843941/+subscriptions

Re: [Qemu-devel] [PATCH v3 1/5] rcu: Add automatically released rcu_read_lock variants

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonz...@redhat.com) wrote:
> On 13/09/19 12:25, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" 
> > 
> > RCU_READ_LOCK_GUARD() takes the rcu_read_lock and then uses glib's
> > g_auto infrastructure (and thus whatever the compiler's hooks are) to
> > release it on all exits of the block.
> > 
> > WITH_RCU_READ_LOCK_GUARD() is similar but is used as a wrapper for the
> > lock, i.e.:
> > 
> >WITH_RCU_READ_LOCK_GUARD() {
> >stuff under lock
> >}
> > 
> > Signed-off-by: Dr. David Alan Gilbert 
> > ---
> >  docs/devel/rcu.txt | 16 
> >  include/qemu/rcu.h | 25 +
> >  2 files changed, 41 insertions(+)
> > 
> > diff --git a/docs/devel/rcu.txt b/docs/devel/rcu.txt
> > index c84e7f42b2..d83fed2f79 100644
> > --- a/docs/devel/rcu.txt
> > +++ b/docs/devel/rcu.txt
> > @@ -187,6 +187,22 @@ The following APIs must be used before RCU is used in 
> > a thread:
> >  Note that these APIs are relatively heavyweight, and should _not_ be
> >  nested.
> >  
> > +Convenience macros
> > +==
> > +
> > +Two macros are provided that automatically release the read lock at the
> > +end of the scope.
> > +
> > +  RCU_READ_LOCK_GUARD()
> > +
> > + Takes the lock and will release it at the end of the block it's
> > + used in.
> > +
> > +  WITH_RCU_READ_LOCK_GUARD()  { code }
> > +
> > + Is used at the head of a block to protect the code within the 
> > block.
> > +
> > +Note that 'goto'ing out of the guarded block will also drop the lock.
> >  
> >  DIFFERENCES WITH LINUX
> >  ==
> > diff --git a/include/qemu/rcu.h b/include/qemu/rcu.h
> > index 22876d1428..3a8d4cf28b 100644
> > --- a/include/qemu/rcu.h
> > +++ b/include/qemu/rcu.h
> > @@ -154,6 +154,31 @@ extern void call_rcu1(struct rcu_head *head, RCUCBFunc 
> > *func);
> >}),\
> >(RCUCBFunc *)g_free);
> >  
> > +typedef void RCUReadAuto;
> > +static inline RCUReadAuto *rcu_read_auto_lock(void)
> > +{
> > +rcu_read_lock();
> > +/* Anything non-NULL causes the cleanup function to be called */
> > +return (void *)(uintptr_t)0x1;
> > +}
> > +
> > +static inline void rcu_read_auto_unlock(RCUReadAuto *r)
> > +{
> > +rcu_read_unlock();
> > +}
> > +
> > +G_DEFINE_AUTOPTR_CLEANUP_FUNC(RCUReadAuto, rcu_read_auto_unlock)
> > +
> > +#define WITH_RCU_READ_LOCK_GUARD() \
> > +WITH_RCU_READ_LOCK_GUARD_(_rcu_read_auto##__COUNTER__)
> > +
> > +#define WITH_RCU_READ_LOCK_GUARD_(var) \
> > +for (g_autoptr(RCUReadAuto) var = rcu_read_auto_lock(); \
> > +(var); rcu_read_auto_unlock(var), (var) = NULL)
> > +
> > +#define RCU_READ_LOCK_GUARD() \
> > +g_autoptr(RCUReadAuto) _rcu_read_auto = rcu_read_auto_lock()
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > 
> 
> Acked-by: Paolo Bonzini 

Can you look at the other bits of this series and tell me if they're OK?
If they are, do you want to take them (since it's RCU based)?

Dave

--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v2] qapi/qmp-dispatch: Fix error class for reporting disabled commands

[Markus Armbruster]

Michal Privoznik  writes:

> On 9/13/19 2:52 PM, Markus Armbruster wrote:
>> Michal Privoznik  writes:
>>
>>> If a command is disabled an error is reported. But due to usage
>>> of error_setg() the class of the error is GenericError which does
>>> not help callers in distinguishing this case from a case where a
>>> qmp command fails regularly due to other reasons. Use
>>> CommandNotFound error class which is much closer to the actual
>>> root cause.
>>>
>>> Signed-off-by: Michal Privoznik 
>>> ---
>>
>> I'd like to tweak the commit message a bit:
>>
>>qmp-dispatch: Use CommandNotFound error for disabled commands
>>
>>If a command is disabled an error is reported.  But due to usage of
>>error_setg() the class of the error is GenericError which does not
>>help callers in distinguishing this case from a case where a qmp
>>command fails regularly due to other reasons.
>>
>>We used to use class CommandDisabled until the great error
>>simplification (commit de253f1491 for QMP and commit 93b91c59db for
>>qemu-ga, both v1.2.0).
>>
>>Use CommandNotFound error class, which is close enough.
>>
>> Objections?
>>
>
> None, thanks for taking care of this.

Need to squash in:

diff --git a/tests/test-qga.c b/tests/test-qga.c
index 891aa3d322..1ca49bbced 100644
--- a/tests/test-qga.c
+++ b/tests/test-qga.c
@@ -668,7 +668,7 @@ static void test_qga_blacklist(gconstpointer data)
 error = qdict_get_qdict(ret, "error");
 class = qdict_get_try_str(error, "class");
 desc = qdict_get_try_str(error, "desc");
-g_assert_cmpstr(class, ==, "GenericError");
+g_assert_cmpstr(class, ==, "CommandNotFound");
 g_assert_nonnull(g_strstr_len(desc, -1, "has been disabled"));
 qobject_unref(ret);
 
@@ -677,7 +677,7 @@ static void test_qga_blacklist(gconstpointer data)
 error = qdict_get_qdict(ret, "error");
 class = qdict_get_try_str(error, "class");
 desc = qdict_get_try_str(error, "desc");
-g_assert_cmpstr(class, ==, "GenericError");
+g_assert_cmpstr(class, ==, "CommandNotFound");
 g_assert_nonnull(g_strstr_len(desc, -1, "has been disabled"));
 qobject_unref(ret);
 

Re: [Qemu-devel] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

[Vladimir Sementsov-Ogievskiy]

and a bit about empty lines

13.09.2019 20:27, Maxim Levitsky wrote:
> Change the qcow2_co_encrypt to just receive full host and
> guest offsets and in pariticular remove the
> offset_in_cluster parameter of do_perform_cow_encrypt,
> since it is misleading, because that offset can be larger than
> cluster size currently.
> 
> Remove the do_perform_cow_encrypt by merging it with
> qcow2_co_encrypt
> 
> Also document the qcow2_co_encrypt arguments to prevent
> that bug from happening again
> 
> Signed-off-by: Maxim Levitsky 
> Reviewed-by: Vladimir Sementsov-Ogievskiy 
> ---

[..]

> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 3b1e63fe41..b31d45fb2b 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
>   }
>   
>   static int coroutine_fn
> -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
> -  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc 
> func)
> +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
> +uint64_t guest_offset, void *buf, size_t len,
> +Qcow2EncDecFunc func)
>   {
>   BDRVQcow2State *s = bs->opaque;
> +

Don't see real benefit in separating one variable initialization from another 
one here,
but I don't mind, it's OK of course.

>   Qcow2EncDecData arg = {
>   .block = s->crypto,
> -.offset = s->crypt_physical_offset ?
> -  file_cluster_offset + offset_into_cluster(s, offset) :
> -  offset,
> +.offset = s->crypt_physical_offset ? host_offset : guest_offset,
>   .buf = buf,
>   .len = len,
>   .func = func,
> @@ -251,18 +251,73 @@ qcow2_co_encdec(BlockDriverState *bs, uint64_t 
> file_cluster_offset,
>   return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
>   }
>   
> +

Hmm, in this file doubled empty lines
are used to separate "Compression" and "Cryptography"
sections. And all other splits are one-empty-line.

> +/*
> + * qcow2_co_encrypt()
> + *
> + * Encrypts one or more contiguous aligned sectors
> + *
> + * @host_offset - underlying storage offset of the first sector of the
> + * data to be encrypted
> + *
> + * @guest_offset - guest (virtual) offset of the first sector of the
> + * data to be encrypted
> + *
> + * @buf - buffer with the data to encrypt, that after encryption
> + *will be written to the underlying storage device at
> + *@host_offset
> + *
> + * @len - length of the buffer (must be a BDRV_SECTOR_SIZE multiple)
> + *
> + * Depending on the encryption method, @host_offset and/or @guest_offset
> + * may be used for generating the initialization vector for
> + * encryption.
> + *
> + * Note that while the whole range must be aligned on sectors, it
> + * does not have to be aligned on clusters and can also cross cluster
> + * boundaries
> + */
>   int coroutine_fn
> -qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
> - uint64_t offset, void *buf, size_t len)
> +qcow2_co_encrypt(BlockDriverState *bs, uint64_t host_offset,
> + uint64_t guest_offset, void *buf, size_t len)
>   {
> -return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
> - qcrypto_block_encrypt);
> +

This empty line is not needed for sure

> +BDRVQcow2State *s = bs->opaque;

But it's common practice to split variables definition from code by empty line 
here

> +assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
> +assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
> +assert(QEMU_IS_ALIGNED(len, BDRV_SECTOR_SIZE));
> +assert(s->crypto);
> +
> +if (!len) {
> +return 0;
> +}
> +
> +return qcow2_co_encdec(bs, host_offset, guest_offset, buf, len,
> +   qcrypto_block_encrypt);
>   }
>   
> +

IMHO extra one (see above)

> +/*
> + * qcow2_co_decrypt()
> + *
> + * Decrypts one or more contiguous aligned sectors
> + * Similar to qcow2_co_encrypt
> + */
> +

Here you replaced one extra empty line by another

>   int coroutine_fn
> -qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
> - uint64_t offset, void *buf, size_t len)
> +qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset,
> + uint64_t guest_offset, void *buf, size_t len)
>   {
> -return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
> - qcrypto_block_decrypt);



-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH] Check correct register for clock source

[Amithash Prasad]

>>  yes. I have pushed it on the aspeed-4.2 branch but it can go independently
>>as there are no conflicts. I changed the title slightly to reflect the
>> area being changed.
Thanks! If required, I can change the patch title and resubmit.

Re: [Qemu-devel] [PATCH v3 6/8] iotests: Test driver whitelisting in 093

[John Snow]

On 9/13/19 8:47 AM, Max Reitz wrote:
> On 20.08.19 23:32, John Snow wrote:
>>
>>
>> On 8/19/19 4:18 PM, Max Reitz wrote:
>>> null-aio may not be whitelisted.  Skip all test cases that require it.
>>>
>>> Signed-off-by: Max Reitz 
>>> ---
>>>  tests/qemu-iotests/093 | 12 +---
>>>  1 file changed, 9 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
>>> index 50c1e7f2ec..f03fa24a07 100755
>>> --- a/tests/qemu-iotests/093
>>> +++ b/tests/qemu-iotests/093
>>> @@ -24,7 +24,7 @@ import iotests
>>>  nsec_per_sec = 10
>>>  
>>>  class ThrottleTestCase(iotests.QMPTestCase):
>>> -test_img = "null-aio://"
>>> +test_driver = "null-aio"
>>>  max_drives = 3
>>>  
>>>  def blockstats(self, device):
>>> @@ -35,10 +35,14 @@ class ThrottleTestCase(iotests.QMPTestCase):
>>>  return stat['rd_bytes'], stat['rd_operations'], 
>>> stat['wr_bytes'], stat['wr_operations']
>>>  raise Exception("Device not found for blockstats: %s" % device)
>>>  
>>> +def required_drivers(self):
>>> +return [self.test_driver]
>>> +
>>> +@iotests.skip_if_unsupported(required_drivers)
>>
>> Oh, I see why you're passing args[0] back to the callback now. Why not
>> just pass self.required_drivers and call it with no arguments instead?
>>
>> You can get a bound version that way that doesn't need additional
>> arguments, and then the callback is free to take generic callables of
>> any kind.
> 
> Am I doing something wrong?
> 
> I just get
> 
> +Traceback (most recent call last):
> +  File "093", line 26, in 
> +class ThrottleTestCase(iotests.QMPTestCase):
> +  File "093", line 41, in ThrottleTestCase
> +@iotests.skip_if_unsupported(self.required_drivers)
> +NameError: name 'self' is not defined
> 
> this way.
> 
> Max
> 
What was I even talking about? :\ Well.

I'd still like to define func_wrapper with a nod to the type constraint
it has:

def func_wrapper(instance: iotests.QMPTestCase, *args, **kwargs):
[...]


Then, you'd write:

if callable(required_formats):
fmts = required_formats(instance)
else:
fmts = required_formats


And:

> +def required_drivers(self):
> +return [self.test_driver]
> +
> +@iotests.skip_if_unsupported(required_drivers)
>  def setUp(self):

The problem is that 'self' isn't defined at the class level, so I was
mistaken about being able to use it :( Python does not have a notion of
a lexical constant to refer to the class being defined; and of course we
do not have an instance variable at definition time.

Sorry for the wild goose chase.

It's fine as-is.

(I wanted a way to define the required_formats callback without forcing
it to be a class method, but the decorator code runs at definition time
and we just don't HAVE the instance; so the way you wrote it is I think
the only way it CAN be written without some much nastier trickery.)

Re: [Qemu-devel] [PATCH v11 04/14] block/backup: introduce BlockCopyState

[Vladimir Sementsov-Ogievskiy]

10.09.2019 13:23, Vladimir Sementsov-Ogievskiy wrote:
> Split copying code part from backup to "block-copy", including separate
> state structure and function renaming. This is needed to share it with
> backup-top filter driver in further commits.
> 
> Notes:
> 
> 1. As BlockCopyState keeps own BlockBackend objects, remaining
> job->common.blk users only use it to get bs by blk_bs() call, so clear
> job->commen.blk permissions set in block_job_create and add
> job->source_bs to be used instead of blk_bs(job->common.blk), to keep
> it more clear which bs we use when introduce backup-top filter in
> further commit.
> 
> 2. Rename s/initializing_bitmap/skip_unallocated/ to sound a bit better
> as interface to BlockCopyState
> 
> 3. Split is not very clean: there left some duplicated fields, backup
> code uses some BlockCopyState fields directly, let's postpone it for
> further improvements and keep this comment simpler for review.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---


[..]

> +
> +static BlockCopyState *block_copy_state_new(
> +BlockDriverState *source, BlockDriverState *target,
> +int64_t cluster_size, BdrvRequestFlags write_flags,
> +ProgressBytesCallbackFunc progress_bytes_callback,
> +ProgressResetCallbackFunc progress_reset_callback,
> +void *progress_opaque, Error **errp)
> +{
> +BlockCopyState *s;
> +int ret;
> +uint64_t no_resize = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE |
> + BLK_PERM_WRITE_UNCHANGED | BLK_PERM_GRAPH_MOD;
> +BdrvDirtyBitmap *copy_bitmap;
> +
> +copy_bitmap = bdrv_create_dirty_bitmap(source, cluster_size, NULL, errp);
> +if (!copy_bitmap) {
> +return NULL;
> +}
> +bdrv_disable_dirty_bitmap(copy_bitmap);
> +
> +s = g_new(BlockCopyState, 1);
> +*s = (BlockCopyState) {
> +.source = blk_new(bdrv_get_aio_context(source),
> +  BLK_PERM_CONSISTENT_READ, no_resize),
> +.target = blk_new(bdrv_get_aio_context(target),
> +  BLK_PERM_WRITE, no_resize),
> +.copy_bitmap = copy_bitmap,
> +.cluster_size = cluster_size,
> +.len = bdrv_dirty_bitmap_size(copy_bitmap),
> +.write_flags = write_flags,
> +.use_copy_range = !(write_flags & BDRV_REQ_WRITE_COMPRESSED),
> +.progress_bytes_callback = progress_bytes_callback,
> +.progress_reset_callback = progress_reset_callback,
> +.progress_opaque = progress_opaque,
> +};
> +
> +s->copy_range_size = QEMU_ALIGN_UP(MIN(blk_get_max_transfer(s->source),
> +   blk_get_max_transfer(s->target)),
> +   s->cluster_size);

preexistent, but it obviously should be QEMU_ALIGN_DOWN. I can resend with a 
separate
fix, it may be fixed while queuing (if resend is not needed for other reasons) 
or
I'll send a follow-up fix later, whichever you prefer.

> +
> +/*
> + * We just allow aio context change on our block backends. block_copy() 
> user
> + * (now it's only backup) is responsible for source and target being in 
> same
> + * aio context.
> + */
> +blk_set_disable_request_queuing(s->source, true);
> +blk_set_allow_aio_context_change(s->source, true);
> +blk_set_disable_request_queuing(s->target, true);
> +blk_set_allow_aio_context_change(s->target, true);
> +

[..]

> @@ -760,21 +860,19 @@ BlockJob *backup_job_create(const char *job_id, 
> BlockDriverState *bs,
>* For more information see commit f8d59dfb40bb and test
>* tests/qemu-iotests/222

[..]

>   job->cluster_size = cluster_size;
> -job->copy_bitmap = copy_bitmap;
> -copy_bitmap = NULL;
> -job->use_copy_range = !compress; /* compression isn't supported for it */
> -job->copy_range_size = 
> MIN_NON_ZERO(blk_get_max_transfer(job->common.blk),
> -blk_get_max_transfer(job->target));
> -job->copy_range_size = MAX(job->cluster_size,
> -   QEMU_ALIGN_UP(job->copy_range_size,
> - job->cluster_size));
> -
> -/* Required permissions are already taken with target's blk_new() */
> +
> +/* Required permissions are already taken by block-copy-state target */
>   block_job_add_bdrv(&job->common, "target", target, 0, BLK_PERM_ALL,
>  &error_abort);
>   job->len = len;


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

[Maxim Levitsky]

On Fri, 2019-09-13 at 17:51 +, Vladimir Sementsov-Ogievskiy wrote:
> 13.09.2019 20:27, Maxim Levitsky wrote:
> > Change the qcow2_co_encrypt to just receive full host and
> > guest offsets and in pariticular remove the
> > offset_in_cluster parameter of do_perform_cow_encrypt,
> > since it is misleading, because that offset can be larger than
> > cluster size currently.
> > 
> > Remove the do_perform_cow_encrypt by merging it with
> > qcow2_co_encrypt
> > 
> > Also document the qcow2_co_encrypt arguments to prevent
> > that bug from happening again
> > 
> > Signed-off-by: Maxim Levitsky 
> > Reviewed-by: Vladimir Sementsov-Ogievskiy 
> > ---
> >   block/qcow2-cluster.c | 35 +--
> >   block/qcow2-threads.c | 81 ---
> >   block/qcow2.c |  5 +--
> >   block/qcow2.h |  8 ++---
> >   4 files changed, 83 insertions(+), 46 deletions(-)
> > 
> > diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> > index bfeb0241d7..f42b8a404c 100644
> > --- a/block/qcow2-cluster.c
> > +++ b/block/qcow2-cluster.c
> > @@ -462,28 +462,6 @@ static int coroutine_fn 
> > do_perform_cow_read(BlockDriverState *bs,
> >   return 0;
> >   }
> >   
> > -static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
> > -uint64_t 
> > src_cluster_offset,
> > -uint64_t cluster_offset,
> > -unsigned offset_in_cluster,
> > -uint8_t *buffer,
> > -unsigned bytes)
> > -{
> > -if (bytes && bs->encrypted) {
> > -BDRVQcow2State *s = bs->opaque;
> > -assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
> > -assert((bytes & ~BDRV_SECTOR_MASK) == 0);
> > -assert(s->crypto);
> > -if (qcow2_co_encrypt(bs,
> > -start_of_cluster(s, cluster_offset + offset_in_cluster),
> > -src_cluster_offset + offset_in_cluster,
> > -buffer, bytes) < 0) {
> > -return false;
> > -}
> > -}
> > -return true;
> > -}
> > -
> >   static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
> >uint64_t cluster_offset,
> >unsigned offset_in_cluster,
> > @@ -891,11 +869,14 @@ static int perform_cow(BlockDriverState *bs, 
> > QCowL2Meta *m)
> >   
> >   /* Encrypt the data if necessary before writing it */
> >   if (bs->encrypted) {
> > -if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> > -start->offset, start_buffer,
> > -start->nb_bytes) ||
> > -!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> > -end->offset, end_buffer, 
> > end->nb_bytes)) {
> > +if (!qcow2_co_encrypt(bs,
> > +  m->offset + start->offset,
> > +  m->alloc_offset + start->offset,
> > +  start_buffer, start->nb_bytes) ||
> 
> You got caught)

Ha!, I am just too tired.

> second argument qcow2_co_encrypt is host_offset and third is guest_offset.
> 
> I noticed this difference when reviewing previous version, and it's actually
> additional reason to drop the function. Unfortunately, I didn't said.
> 
> Also, we can use return value of qcow2_co_encrypt instead of dropping it,
> 
> something like classical
> 
> ret = qcow2_co_encrypt(..)
> if (ret < 0) {
> goto fail;
> }
> 
> ret = qcow2_co_encrypt(..)
> if (ret < 0) {
> goto fail;
> }
> 
> Also, IMHO, too much changes too keep reviewed-by mark. Still, maybe it's my
> "with it least fixed" is misleading, may be interpreted like removing the
> function and corresponding refactoring is included..
Yea. So V7 there will be...
Re-factoring is hard (but worth it) :-(

> 
> 
> > +!qcow2_co_encrypt(bs,
> > +  m->offset + end->offset,
> > +  m->alloc_offset + end->offset,
> > +  end_buffer, end->nb_bytes)) {
> >   ret = -EIO;
> >   goto fail;
> >   }
> > diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> > index 3b1e63fe41..b31d45fb2b 100644
> > --- a/block/qcow2-threads.c
> > +++ b/block/qcow2-threads.c
> > @@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
> >   }
> >   
> >   static int coroutine_fn
> > -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
> > -  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc 
> > func)
> > +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
> > +uint64_t guest_offset, void *buf, size_t len,
> > +Qcow2EncDecFu

Re: [Qemu-devel] [PATCH 0/2] HPPA tcg fixes

[Richard Henderson]

On 9/13/19 6:17 AM, Sven Schnelle wrote:
> Sven Schnelle (2):
>   target/hppa: prevent trashing of temporary in trans_mtctl()
>   target/hppa: prevent trashing of temporary in do_depw_sar()
> 
>  target/hppa/translate.c | 15 ++-
>  1 file changed, 10 insertions(+), 5 deletions(-)

Thanks, queued.


r~

Re: [Qemu-devel] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

[Vladimir Sementsov-Ogievskiy]

13.09.2019 20:27, Maxim Levitsky wrote:
> Change the qcow2_co_encrypt to just receive full host and
> guest offsets and in pariticular remove the
> offset_in_cluster parameter of do_perform_cow_encrypt,
> since it is misleading, because that offset can be larger than
> cluster size currently.
> 
> Remove the do_perform_cow_encrypt by merging it with
> qcow2_co_encrypt
> 
> Also document the qcow2_co_encrypt arguments to prevent
> that bug from happening again
> 
> Signed-off-by: Maxim Levitsky 
> Reviewed-by: Vladimir Sementsov-Ogievskiy 
> ---
>   block/qcow2-cluster.c | 35 +--
>   block/qcow2-threads.c | 81 ---
>   block/qcow2.c |  5 +--
>   block/qcow2.h |  8 ++---
>   4 files changed, 83 insertions(+), 46 deletions(-)
> 
> diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> index bfeb0241d7..f42b8a404c 100644
> --- a/block/qcow2-cluster.c
> +++ b/block/qcow2-cluster.c
> @@ -462,28 +462,6 @@ static int coroutine_fn 
> do_perform_cow_read(BlockDriverState *bs,
>   return 0;
>   }
>   
> -static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
> -uint64_t src_cluster_offset,
> -uint64_t cluster_offset,
> -unsigned offset_in_cluster,
> -uint8_t *buffer,
> -unsigned bytes)
> -{
> -if (bytes && bs->encrypted) {
> -BDRVQcow2State *s = bs->opaque;
> -assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
> -assert((bytes & ~BDRV_SECTOR_MASK) == 0);
> -assert(s->crypto);
> -if (qcow2_co_encrypt(bs,
> -start_of_cluster(s, cluster_offset + offset_in_cluster),
> -src_cluster_offset + offset_in_cluster,
> -buffer, bytes) < 0) {
> -return false;
> -}
> -}
> -return true;
> -}
> -
>   static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
>uint64_t cluster_offset,
>unsigned offset_in_cluster,
> @@ -891,11 +869,14 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta 
> *m)
>   
>   /* Encrypt the data if necessary before writing it */
>   if (bs->encrypted) {
> -if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> -start->offset, start_buffer,
> -start->nb_bytes) ||
> -!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> -end->offset, end_buffer, end->nb_bytes)) 
> {
> +if (!qcow2_co_encrypt(bs,
> +  m->offset + start->offset,
> +  m->alloc_offset + start->offset,
> +  start_buffer, start->nb_bytes) ||

You got caught)
second argument qcow2_co_encrypt is host_offset and third is guest_offset.

I noticed this difference when reviewing previous version, and it's actually
additional reason to drop the function. Unfortunately, I didn't said.

Also, we can use return value of qcow2_co_encrypt instead of dropping it,

something like classical

ret = qcow2_co_encrypt(..)
if (ret < 0) {
goto fail;
}

ret = qcow2_co_encrypt(..)
if (ret < 0) {
goto fail;
}

Also, IMHO, too much changes too keep reviewed-by mark. Still, maybe it's my
"with it least fixed" is misleading, may be interpreted like removing the
function and corresponding refactoring is included..


> +!qcow2_co_encrypt(bs,
> +  m->offset + end->offset,
> +  m->alloc_offset + end->offset,
> +  end_buffer, end->nb_bytes)) {
>   ret = -EIO;
>   goto fail;
>   }
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 3b1e63fe41..b31d45fb2b 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
>   }
>   
>   static int coroutine_fn
> -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
> -  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc 
> func)
> +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
> +uint64_t guest_offset, void *buf, size_t len,
> +Qcow2EncDecFunc func)
>   {
>   BDRVQcow2State *s = bs->opaque;
> +
>   Qcow2EncDecData arg = {
>   .block = s->crypto,
> -.offset = s->crypt_physical_offset ?
> -  file_cluster_offset + offset_into_cluster(s, offset) :
> -  offset,
> +.offset = s->crypt_physical_offset ? host_offset : guest_offset,
>   .buf = buf,
>   .len = len,
>

[Qemu-devel] [PATCH v6 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[Maxim Levitsky]

This fixes subtle corruption introduced by luks threaded encryption
in commit 8ac0f15f335

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922

The corruption happens when we do a write that
   * writes to two or more unallocated clusters at once
   * doesn't fully cover the first sector
   * doesn't fully cover the last sector

In this case, when allocating the new clusters we COW both areas
prior to the write and after the write, and we encrypt them.

The above mentioned commit accidentally made it so we encrypt the
second COW area using the physical cluster offset of the first area.

The problem is that offset_in_cluster in do_perform_cow_encrypt
can be larger that the cluster size, thus cluster_offset
will no longer point to the start of the cluster at which encrypted
area starts.

Next patch in this series will refactor the code to avoid all these
assumptions.

In the bugreport that was triggered by rebasing a luks image to new,
zero filled base, which lot of such writes, and causes some files
with zero areas to contain garbage there instead.
But as described above it can happen elsewhere as well


Signed-off-by: Maxim Levitsky 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-cluster.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index dcacd3c450..bfeb0241d7 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -474,9 +474,10 @@ static bool coroutine_fn 
do_perform_cow_encrypt(BlockDriverState *bs,
 assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
 assert((bytes & ~BDRV_SECTOR_MASK) == 0);
 assert(s->crypto);
-if (qcow2_co_encrypt(bs, cluster_offset,
- src_cluster_offset + offset_in_cluster,
- buffer, bytes) < 0) {
+if (qcow2_co_encrypt(bs,
+start_of_cluster(s, cluster_offset + offset_in_cluster),
+src_cluster_offset + offset_in_cluster,
+buffer, bytes) < 0) {
 return false;
 }
 }
-- 
2.17.2

[Qemu-devel] [PATCH v6 3/3] qemu-iotests: Add test for bz #1745922

[Maxim Levitsky]

Signed-off-by: Maxim Levitsky 
Tested-by: Vladimir Sementsov-Ogievskiy 
---
 tests/qemu-iotests/263 | 91 ++
 tests/qemu-iotests/263.out | 40 +
 tests/qemu-iotests/group   |  1 +
 3 files changed, 132 insertions(+)
 create mode 100755 tests/qemu-iotests/263
 create mode 100644 tests/qemu-iotests/263.out

diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
new file mode 100755
index 00..d2c030fae9
--- /dev/null
+++ b/tests/qemu-iotests/263
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# Test encrypted write that crosses cluster boundary of two unallocated 
clusters
+# Based on 188
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=mlevi...@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=1M
+
+SECRET="secret,id=sec0,data=astrochicken"
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+
+_run_test()
+{
+   echo "== reading the whole image =="
+   $QEMU_IO --object $SECRET -c "read -P 0 0 $size" --image-opts "$1" | 
_filter_qemu_io | _filter_testdir
+
+   echo
+   echo "== write two 512 byte sectors on a cluster boundary =="
+   $QEMU_IO --object $SECRET -c "write -P 0xAA 0xFE00 0x400" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+
+   echo
+   echo "== verify that the rest of the image is not changed =="
+   $QEMU_IO --object $SECRET -c "read -P 0x00 0x0 0xFE00" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+   $QEMU_IO --object $SECRET -c "read -P 0xAA 0x0FE00 0x400" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+   $QEMU_IO --object $SECRET -c "read -P 0x00 0x10200 0xEFE00" 
--image-opts "$1" | _filter_qemu_io | _filter_testdir
+
+}
+
+
+echo
+echo "testing LUKS qcow2 encryption"
+echo
+
+_make_test_img --object $SECRET -o 
"encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=64K"
 $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+echo
+echo "testing legacy AES qcow2 encryption"
+echo
+
+
+_make_test_img --object $SECRET -o 
"encrypt.format=aes,encrypt.key-secret=sec0,cluster_size=64K" $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/263.out b/tests/qemu-iotests/263.out
new file mode 100644
index 00..0c982c55cb
--- /dev/null
+++ b/tests/qemu-iotests/263.out
@@ -0,0 +1,40 @@
+QA output created by 263
+
+testing LUKS qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+testing legacy AES qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=aes 
encrypt.key-secret=sec0
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.50

[Qemu-devel] [PATCH v6 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[Maxim Levitsky]

Commit 8ac0f15f335 accidently broke the COW of non changed areas
of newly allocated clusters, when the write spans multiple clusters,
and needs COW both prior and after the write.
This results in 'after' COW area being encrypted with wrong
sector address, which render it corrupted.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922

CC: qemu-stable 

V2: grammar, spelling and code style fixes.
V3: more fixes after the review.
V4: addressed review comments from Max Reitz,
and futher refactored the qcow2_co_encrypt to just take full host and guest 
offset
which simplifies everything.

V5: reworked the patches so one of them fixes the bug
only and other one is just refactoring

V6: removed do_perform_cow_encrypt

Best regards,
Maxim Levitsky

Maxim Levitsky (3):
  Fix qcow2+luks corruption introduced by commit 8ac0f15f335
  block/qcow2: refactor threaded encryption code
  qemu-iotests: Add test for bz #1745922

 block/qcow2-cluster.c  | 34 --
 block/qcow2-threads.c  | 81 +++--
 block/qcow2.c  |  5 ++-
 block/qcow2.h  |  8 ++--
 tests/qemu-iotests/263 | 91 ++
 tests/qemu-iotests/263.out | 40 +
 tests/qemu-iotests/group   |  1 +
 7 files changed, 215 insertions(+), 45 deletions(-)
 create mode 100755 tests/qemu-iotests/263
 create mode 100644 tests/qemu-iotests/263.out

-- 
2.17.2

[Qemu-devel] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

[Maxim Levitsky]

Change the qcow2_co_encrypt to just receive full host and
guest offsets and in pariticular remove the
offset_in_cluster parameter of do_perform_cow_encrypt,
since it is misleading, because that offset can be larger than
cluster size currently.

Remove the do_perform_cow_encrypt by merging it with
qcow2_co_encrypt

Also document the qcow2_co_encrypt arguments to prevent
that bug from happening again

Signed-off-by: Maxim Levitsky 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-cluster.c | 35 +--
 block/qcow2-threads.c | 81 ---
 block/qcow2.c |  5 +--
 block/qcow2.h |  8 ++---
 4 files changed, 83 insertions(+), 46 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index bfeb0241d7..f42b8a404c 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -462,28 +462,6 @@ static int coroutine_fn 
do_perform_cow_read(BlockDriverState *bs,
 return 0;
 }
 
-static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
-uint64_t src_cluster_offset,
-uint64_t cluster_offset,
-unsigned offset_in_cluster,
-uint8_t *buffer,
-unsigned bytes)
-{
-if (bytes && bs->encrypted) {
-BDRVQcow2State *s = bs->opaque;
-assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
-assert((bytes & ~BDRV_SECTOR_MASK) == 0);
-assert(s->crypto);
-if (qcow2_co_encrypt(bs,
-start_of_cluster(s, cluster_offset + offset_in_cluster),
-src_cluster_offset + offset_in_cluster,
-buffer, bytes) < 0) {
-return false;
-}
-}
-return true;
-}
-
 static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
  uint64_t cluster_offset,
  unsigned offset_in_cluster,
@@ -891,11 +869,14 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta 
*m)
 
 /* Encrypt the data if necessary before writing it */
 if (bs->encrypted) {
-if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-start->offset, start_buffer,
-start->nb_bytes) ||
-!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-end->offset, end_buffer, end->nb_bytes)) {
+if (!qcow2_co_encrypt(bs,
+  m->offset + start->offset,
+  m->alloc_offset + start->offset,
+  start_buffer, start->nb_bytes) ||
+!qcow2_co_encrypt(bs,
+  m->offset + end->offset,
+  m->alloc_offset + end->offset,
+  end_buffer, end->nb_bytes)) {
 ret = -EIO;
 goto fail;
 }
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 3b1e63fe41..b31d45fb2b 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
 }
 
 static int coroutine_fn
-qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
-  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc func)
+qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
+uint64_t guest_offset, void *buf, size_t len,
+Qcow2EncDecFunc func)
 {
 BDRVQcow2State *s = bs->opaque;
+
 Qcow2EncDecData arg = {
 .block = s->crypto,
-.offset = s->crypt_physical_offset ?
-  file_cluster_offset + offset_into_cluster(s, offset) :
-  offset,
+.offset = s->crypt_physical_offset ? host_offset : guest_offset,
 .buf = buf,
 .len = len,
 .func = func,
@@ -251,18 +251,73 @@ qcow2_co_encdec(BlockDriverState *bs, uint64_t 
file_cluster_offset,
 return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
 }
 
+
+/*
+ * qcow2_co_encrypt()
+ *
+ * Encrypts one or more contiguous aligned sectors
+ *
+ * @host_offset - underlying storage offset of the first sector of the
+ * data to be encrypted
+ *
+ * @guest_offset - guest (virtual) offset of the first sector of the
+ * data to be encrypted
+ *
+ * @buf - buffer with the data to encrypt, that after encryption
+ *will be written to the underlying storage device at
+ *@host_offset
+ *
+ * @len - length of the buffer (must be a BDRV_SECTOR_SIZE multiple)
+ *
+ * Depending on the encryption method, @host_offset and/or @guest_offset
+ * may be used for generating the initialization vector for
+ * encryption.
+ *
+ * Note that while the whole range must be aligned on sectors, it
+ * does not hav

Re: [Qemu-devel] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

[Maxim Levitsky]

On Fri, 2019-09-13 at 16:57 +, Vladimir Sementsov-Ogievskiy wrote:
> 13.09.2019 19:39, Maxim Levitsky wrote:
> > On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote:
> > > 13.09.2019 18:28, Maxim Levitsky wrote:
> > > > Signed-off-by: Maxim Levitsky 
> > > > ---
> > > >tests/qemu-iotests/263 | 91 
> > > > ++
> > > >tests/qemu-iotests/263.out | 40 +
> > > >tests/qemu-iotests/group   |  2 +
> > > >3 files changed, 133 insertions(+)
> > > >create mode 100755 tests/qemu-iotests/263
> > > >create mode 100644 tests/qemu-iotests/263.out
> > > > 
> > > > diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
> > > > new file mode 100755
> > > > index 00..d2c030fae9
> > > > --- /dev/null
> > > > +++ b/tests/qemu-iotests/263
> > > 
> > > [..]
> > > 
> > > > --- a/tests/qemu-iotests/group
> > > > +++ b/tests/qemu-iotests/group
> > > > @@ -274,5 +274,7 @@
> > > >257 rw
> > > >258 rw quick
> > > >262 rw quick migration
> > > > +263 rw quick
> > > >265 rw auto quick
> > > >266 rw quick
> > > > +>>> patched
> > > > 
> > > 
> > > last line is a mistake.
> > 
> > Last minute merge mistake :-(
> > 
> > > 
> > > also, test failed for me:
> > > 
> > > 263  fail   [19:21:35] [19:21:35]output 
> > > mismatch (see 263.out.bad)
> > > --- /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out
> > > 2019-09-13 19:11:45.464727427 +0300
> > > +++ /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out.bad
> > > 2019-09-13 19:21:35.535381253 +0300
> > > @@ -2,6 +2,7 @@
> > > 
> > >testing LUKS qcow2 encryption
> > > 
> > > +qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this 
> > > build: Function not implemented
> > >Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 
> > > encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
> > >== reading the whole image ==
> > >read 1048576/1048576 bytes at offset 0
> > > Failures: 263
> > > Failed 1 of 1 tests
> > > 
> > > 
> > > and if reconfigure with
> > >--enable-gnutls --enable-gcrypt --enable-vhost-crypto 
> > > --enable-crypto-afalg
> > > (don't know which one is actually needed)
> > > it works..
> > > 
> > > so, we at least should skip the test if it's unsupported
> > 
> > Don't know. I based this test on 188 and it also doesn't have anything 
> > special about this.
> > Its not in auto group though.
> > I guess we need to ask Daniel about this.
> > 
> > 
> 
> Hmm, the problem was not in "configure" arguments, seems needed thing is 
> enabled by default, but actually
> all works for me after installing gnutls-devel
> 
> and without it 188 don't work without it (or some its dependencies) too. So 
> it's "not a regression"
> and I'dont care:
> 
> Tested-by: Vladimir Sementsov-Ogievskiy 
>[don't forget to drop ">>> patched"]
> 
> and I also checked that test fails if position it before patch 01, so it's 
> true bug-catcher.

Thanks!!

Best regards,
Maxim Levitsky

Re: [Qemu-devel] [PATCH v7 0/3] 9p: Fix file ID collisions

[Greg Kurz]

On Thu, 5 Sep 2019 12:42:01 +0200
Christian Schoenebeck  wrote:

> This is v7 of a proposed patch set for fixing file ID collisions with 9pfs.
> 

So I did some changes in 1/3 and pushed everything to 9p-next. I'll do some
more manual testing and issue a PR when I'm confident enough.

It would be nice to have some sort of automated test for that in 'make check'.
My first thought is to simulate a cross-device setup with the synth backend,
because it might be difficult to do this on a real filesystem without requiring
elevated privileges.

> v6->v7:
> 
>   * Rebased to https://github.com/gkurz/qemu/commits/9p-next
> (SHA1 7fc4c49e91).
> 
>   * Be pedantic and abort with error on wrong value for new command line
> argument 'multidevs'.
> 
>   * Adjusted patches to qemu code style guidelines.
> 
>   * Fixed potential crash in qp_table_destroy() on error path.
> 
>   * Use dedicated hash table init functions (qpd_table_init(),
> qpf_table_init(), qpp_table_init()):
> https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00144.html
> 
>   * Use warn_report_once() instead of error_report_once() for issues
> interpreted merely as being warnings, not errors.
> 
>   * Simplified hash table destruction (due to simplified error path
> introduced by SHA1 7fc4c49e91).
> 
>   * Dropped capturing root_ino for now:
> https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00146.html
> 
>   * Don't update proxy docs, since new command line argument 'multidevs' is
> limited to the local backend for now.
> 
>   * Mention in docs that readdir() is currently not blocked by
> 'multidevs=forbid'.
> 
>   * Rename qid_path_prefixmap() -> qid_path_suffixmap() in patch 3
> (due to the semantic change of that function by that patch).
> 
> Christian Schoenebeck (3):
>   9p: Added virtfs option 'multidevs=remap|forbid|warn'
>   9p: stat_to_qid: implement slow path
>   9p: Use variable length suffixes for inode remapping
> 
>  fsdev/file-op-9p.h  |   5 +
>  fsdev/qemu-fsdev-opts.c |   7 +-
>  fsdev/qemu-fsdev.c  |  17 ++
>  hw/9pfs/9p.c| 456 ++--
>  hw/9pfs/9p.h|  59 ++
>  qemu-options.hx |  26 ++-
>  vl.c|   7 +-
>  7 files changed, 552 insertions(+), 25 deletions(-)
> 

Re: [Qemu-devel] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

[Vladimir Sementsov-Ogievskiy]

13.09.2019 19:39, Maxim Levitsky wrote:
> On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote:
>> 13.09.2019 18:28, Maxim Levitsky wrote:
>>> Signed-off-by: Maxim Levitsky 
>>> ---
>>>tests/qemu-iotests/263 | 91 ++
>>>tests/qemu-iotests/263.out | 40 +
>>>tests/qemu-iotests/group   |  2 +
>>>3 files changed, 133 insertions(+)
>>>create mode 100755 tests/qemu-iotests/263
>>>create mode 100644 tests/qemu-iotests/263.out
>>>
>>> diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
>>> new file mode 100755
>>> index 00..d2c030fae9
>>> --- /dev/null
>>> +++ b/tests/qemu-iotests/263
>>
>> [..]
>>
>>> --- a/tests/qemu-iotests/group
>>> +++ b/tests/qemu-iotests/group
>>> @@ -274,5 +274,7 @@
>>>257 rw
>>>258 rw quick
>>>262 rw quick migration
>>> +263 rw quick
>>>265 rw auto quick
>>>266 rw quick
>>> +>>> patched
>>>
>>
>> last line is a mistake.
> Last minute merge mistake :-(
> 
>>
>> also, test failed for me:
>>
>> 263  fail   [19:21:35] [19:21:35]output mismatch 
>> (see 263.out.bad)
>> --- /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out2019-09-13 
>> 19:11:45.464727427 +0300
>> +++ /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out.bad2019-09-13 
>> 19:21:35.535381253 +0300
>> @@ -2,6 +2,7 @@
>>
>>testing LUKS qcow2 encryption
>>
>> +qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this 
>> build: Function not implemented
>>Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 
>> encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
>>== reading the whole image ==
>>read 1048576/1048576 bytes at offset 0
>> Failures: 263
>> Failed 1 of 1 tests
>>
>>
>> and if reconfigure with
>>--enable-gnutls --enable-gcrypt --enable-vhost-crypto 
>> --enable-crypto-afalg
>> (don't know which one is actually needed)
>> it works..
>>
>> so, we at least should skip the test if it's unsupported
> Don't know. I based this test on 188 and it also doesn't have anything 
> special about this.
> Its not in auto group though.
> I guess we need to ask Daniel about this.
> 
> 

Hmm, the problem was not in "configure" arguments, seems needed thing is 
enabled by default, but actually
all works for me after installing gnutls-devel

and without it 188 don't work without it (or some its dependencies) too. So 
it's "not a regression"
and I'dont care:

Tested-by: Vladimir Sementsov-Ogievskiy 
   [don't forget to drop ">>> patched"]

and I also checked that test fails if position it before patch 01, so it's true 
bug-catcher.

-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

[Maxim Levitsky]

On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote:
> 13.09.2019 18:28, Maxim Levitsky wrote:
> > Signed-off-by: Maxim Levitsky 
> > ---
> >   tests/qemu-iotests/263 | 91 ++
> >   tests/qemu-iotests/263.out | 40 +
> >   tests/qemu-iotests/group   |  2 +
> >   3 files changed, 133 insertions(+)
> >   create mode 100755 tests/qemu-iotests/263
> >   create mode 100644 tests/qemu-iotests/263.out
> > 
> > diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
> > new file mode 100755
> > index 00..d2c030fae9
> > --- /dev/null
> > +++ b/tests/qemu-iotests/263
> 
> [..]
> 
> > --- a/tests/qemu-iotests/group
> > +++ b/tests/qemu-iotests/group
> > @@ -274,5 +274,7 @@
> >   257 rw
> >   258 rw quick
> >   262 rw quick migration
> > +263 rw quick
> >   265 rw auto quick
> >   266 rw quick
> > +>>> patched
> > 
> 
> last line is a mistake.
Last minute merge mistake :-(

> 
> also, test failed for me:
> 
> 263  fail   [19:21:35] [19:21:35]output mismatch 
> (see 263.out.bad)
> --- /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out2019-09-13 
> 19:11:45.464727427 +0300
> +++ /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out.bad2019-09-13 
> 19:21:35.535381253 +0300
> @@ -2,6 +2,7 @@
> 
>   testing LUKS qcow2 encryption
> 
> +qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this 
> build: Function not implemented
>   Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks 
> encrypt.key-secret=sec0 encrypt.iter-time=10
>   == reading the whole image ==
>   read 1048576/1048576 bytes at offset 0
> Failures: 263
> Failed 1 of 1 tests
> 
> 
> and if reconfigure with
>   --enable-gnutls --enable-gcrypt --enable-vhost-crypto --enable-crypto-afalg
> (don't know which one is actually needed)
> it works..
> 
> so, we at least should skip the test if it's unsupported
Don't know. I based this test on 188 and it also doesn't have anything special 
about this.
Its not in auto group though.
I guess we need to ask Daniel about this.



Best regards,
Maxim Levitsky

[Qemu-devel] [PATCH 1/2] migration/rdma: Don't moan about disconnects at the end

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

If we've already finished the migration or something has
already gone wrong, don't moan about the migration stream disconnecting.

Signed-off-by: Dr. David Alan Gilbert 
---
 migration/rdma.c | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 78e6b72bac..0fcf02f48e 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3253,10 +3253,14 @@ static void rdma_cm_poll_handler(void *opaque)
 
 if (cm_event->event == RDMA_CM_EVENT_DISCONNECTED ||
 cm_event->event == RDMA_CM_EVENT_DEVICE_REMOVAL) {
-error_report("receive cm event, cm event is %d", cm_event->event);
-rdma->error_state = -EPIPE;
-if (rdma->return_path) {
-rdma->return_path->error_state = -EPIPE;
+if (!rdma->error_state &&
+migration_incoming_get_current()->state !=
+  MIGRATION_STATUS_COMPLETED) {
+error_report("receive cm event, cm event is %d", cm_event->event);
+rdma->error_state = -EPIPE;
+if (rdma->return_path) {
+rdma->return_path->error_state = -EPIPE;
+}
 }
 
 if (mis->migration_incoming_co) {
-- 
2.21.0

[Qemu-devel] [PATCH] 9p: Print error hints if option parsing fails

[Greg Kurz]

Option parsing fonctions are called with &error_fatal, which
causes error_setg() to call exit() and the hints are never
printed.

Use an intermediate error object so that exit() happens in
error_propagate() after error_append_hint() could be called.

Signed-off-by: Greg Kurz 
---
 hw/9pfs/9p-local.c |   11 +++
 hw/9pfs/9p-proxy.c |   11 +++
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/hw/9pfs/9p-local.c b/hw/9pfs/9p-local.c
index 6f7309f4e691..2c80d2ac30f5 100644
--- a/hw/9pfs/9p-local.c
+++ b/hw/9pfs/9p-local.c
@@ -1486,8 +1486,9 @@ static int local_parse_opts(QemuOpts *opts, FsDriverEntry 
*fse, Error **errp)
 Error *local_err = NULL;
 
 if (!sec_model) {
-error_setg(errp, "security_model property not set");
-error_append_security_model_hint(errp);
+error_setg(&local_err, "security_model property not set");
+error_append_security_model_hint(&local_err);
+error_propagate(errp, local_err);
 return -1;
 }
 
@@ -1501,8 +1502,10 @@ static int local_parse_opts(QemuOpts *opts, 
FsDriverEntry *fse, Error **errp)
 } else if (!strcmp(sec_model, "mapped-file")) {
 fse->export_flags |= V9FS_SM_MAPPED_FILE;
 } else {
-error_setg(errp, "invalid security_model property '%s'", sec_model);
-error_append_security_model_hint(errp);
+error_setg(&local_err, "invalid security_model property '%s'",
+   sec_model);
+error_append_security_model_hint(&local_err);
+error_propagate(errp, local_err);
 return -1;
 }
 
diff --git a/hw/9pfs/9p-proxy.c b/hw/9pfs/9p-proxy.c
index 97ab9c58a573..5de5e53702f7 100644
--- a/hw/9pfs/9p-proxy.c
+++ b/hw/9pfs/9p-proxy.c
@@ -1126,15 +1126,18 @@ static int proxy_parse_opts(QemuOpts *opts, 
FsDriverEntry *fs, Error **errp)
 {
 const char *socket = qemu_opt_get(opts, "socket");
 const char *sock_fd = qemu_opt_get(opts, "sock_fd");
+Error *local_err = NULL;
 
 if (!socket && !sock_fd) {
-error_setg(errp, "both socket and sock_fd properties are missing");
-error_append_socket_sockfd_hint(errp);
+error_setg(&local_err, "both socket and sock_fd properties are 
missing");
+error_append_socket_sockfd_hint(&local_err);
+error_propagate(errp, local_err);
 return -1;
 }
 if (socket && sock_fd) {
-error_setg(errp, "both socket and sock_fd properties are set");
-error_append_socket_sockfd_hint(errp);
+error_setg(&local_err, "both socket and sock_fd properties are set");
+error_append_socket_sockfd_hint(&local_err);
+error_propagate(errp, local_err);
 return -1;
 }
 if (socket) {

[Qemu-devel] [PATCH 0/2] migration/rdma disconnect fixes

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

Hi,
  This fixes a deadlock that can occur on the source after
a failed RDMA migration and cleans up some warning messages
that can appear during normal completion.

https://bugzilla.redhat.com/show_bug.cgi?id=1746787

Dr. David Alan Gilbert (2):
  migration/rdma: Don't moan about disconnects at the end
  migration/rdma.c: Swap synchronize_rcu for call_rcu

 migration/rdma.c | 51 +---
 1 file changed, 35 insertions(+), 16 deletions(-)

-- 
2.21.0

[Qemu-devel] [PATCH 2/2] migration/rdma.c: Swap synchronize_rcu for call_rcu

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

This fixes a deadlock that can occur on the migration source after
a failed RDMA migration;  as the source tries to cleanup it
clears a pair of pointers and uses synchronize_rcu to wait; this
is happening on the main thread.  With the CPUs running
a CPU thread can be an rcu reader and attempt to grab the main lock
(kvm_handle_io->address_space_write->flatview_write->flatview_write_continue->
prepare_mmio_access->qemu_mutex_lock_iothread_impl)

Replace the synchronize_rcu with a call_rcu to postpone the freeing.

Fixes: 74637e6f08fceda98806 ("migration: implement bi-directional RDMA 
QIOChannel")

( https://bugzilla.redhat.com/show_bug.cgi?id=1746787 )

Signed-off-by: Dr. David Alan Gilbert 
---
 migration/rdma.c | 39 +++
 1 file changed, 27 insertions(+), 12 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 0fcf02f48e..4c74e88a37 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3017,11 +3017,35 @@ static void 
qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc,
 }
 }
 
+struct rdma_close_rcu {
+struct rcu_head rcu;
+RDMAContext *rdmain;
+RDMAContext *rdmaout;
+};
+
+/* callback from qio_channel_rdma_close via call_rcu */
+static void qio_channel_rdma_close_rcu(struct rdma_close_rcu *rcu)
+{
+if (rcu->rdmain) {
+qemu_rdma_cleanup(rcu->rdmain);
+}
+
+if (rcu->rdmaout) {
+qemu_rdma_cleanup(rcu->rdmaout);
+}
+
+g_free(rcu->rdmain);
+g_free(rcu->rdmaout);
+g_free(rcu);
+}
+
 static int qio_channel_rdma_close(QIOChannel *ioc,
   Error **errp)
 {
 QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
 RDMAContext *rdmain, *rdmaout;
+struct rdma_close_rcu *rcu = g_new(struct rdma_close_rcu, 1);
+
 trace_qemu_rdma_close();
 
 rdmain = rioc->rdmain;
@@ -3034,18 +3058,9 @@ static int qio_channel_rdma_close(QIOChannel *ioc,
 atomic_rcu_set(&rioc->rdmaout, NULL);
 }
 
-synchronize_rcu();
-
-if (rdmain) {
-qemu_rdma_cleanup(rdmain);
-}
-
-if (rdmaout) {
-qemu_rdma_cleanup(rdmaout);
-}
-
-g_free(rdmain);
-g_free(rdmaout);
+rcu->rdmain = rdmain;
+rcu->rdmaout = rdmaout;
+call_rcu(rcu, qio_channel_rdma_close_rcu, rcu);
 
 return 0;
 }
-- 
2.21.0

Re: [Qemu-devel] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

[Maxim Levitsky]

On Fri, 2019-09-13 at 16:11 +, Vladimir Sementsov-Ogievskiy wrote:
> 13.09.2019 18:28, Maxim Levitsky wrote:
> > Change do_perform_cow_encrypt and its callee qcow2_co_encrypt
> > to just receive full host and guest offsets and in pariticular
> > remove the offset_in_cluster parameter of do_perform_cow_encrypt,
> > since it is misleading, because that offset can be larger than
> > cluster size currently.
> > 
> > Also document the qcow2_co_encrypt arguments to prevent
> > that bug from happening again
> > 
> > Signed-off-by: Maxim Levitsky 
> > ---
> >   block/qcow2-cluster.c | 30 -
> >   block/qcow2-threads.c | 62 ++-
> >   block/qcow2.c |  5 ++--
> >   block/qcow2.h |  8 +++---
> >   4 files changed, 73 insertions(+), 32 deletions(-)
> > 
> > diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> > index bfeb0241d7..e87a4637fd 100644
> > --- a/block/qcow2-cluster.c
> > +++ b/block/qcow2-cluster.c
> > @@ -463,21 +463,21 @@ static int coroutine_fn 
> > do_perform_cow_read(BlockDriverState *bs,
> >   }
> >   
> >   static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
> > -uint64_t 
> > src_cluster_offset,
> > -uint64_t cluster_offset,
> > -unsigned offset_in_cluster,
> > +uint64_t guest_offset,
> > +uint64_t host_offset,
> >   uint8_t *buffer,
> >   unsigned bytes)
> >   {
> >   if (bytes && bs->encrypted) {
> >   BDRVQcow2State *s = bs->opaque;
> > -assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
> > -assert((bytes & ~BDRV_SECTOR_MASK) == 0);
> > +
> > +assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
> > +assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
> > +assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
> >   assert(s->crypto);
> > -if (qcow2_co_encrypt(bs,
> > -start_of_cluster(s, cluster_offset + offset_in_cluster),
> > -src_cluster_offset + offset_in_cluster,
> > -buffer, bytes) < 0) {
> > +
> > +if (qcow2_co_encrypt(bs, host_offset, guest_offset,
> > +buffer, bytes) < 0) {
> 
> strange alignment of second line of the condition.. [1]
To be honest I am too tired currently, I think I should align to the (
of function argument if possible.
I removed the do_perform_cow_encrypt as you suggested so no issue.

> 
> >   return false;
> >   }
> >   }
> > @@ -891,11 +891,15 @@ static int perform_cow(BlockDriverState *bs, 
> > QCowL2Meta *m)
> >   
> >   /* Encrypt the data if necessary before writing it */
> >   if (bs->encrypted) {
> > -if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> > -start->offset, start_buffer,
> > +if (!do_perform_cow_encrypt(bs,
> > +m->offset + start->offset,
> > +m->alloc_offset + start->offset,
> > +start_buffer,
> >   start->nb_bytes) ||
> > -!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> > -end->offset, end_buffer, 
> > end->nb_bytes)) {
> > +!do_perform_cow_encrypt(bs,
> > +m->offset + end->offset,
> > +m->alloc_offset + end->offset,
> > +end_buffer, end->nb_bytes)) {
> 
> Looking at this now, I think that do_perform_cow_encrypt can be dropped at 
> all,
> as it's now just an extra wrapper, sending same parameters to 
> qcow2_co_encrypt.
> 
> I'll send a follow-up, if this patch goes as is.
I kind of agree but I didn't want to make too many changes. I do think 
now also that it is worth it.


> 
> >   ret = -EIO;
> >   goto fail;
> >   }
> > diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> > index 3b1e63fe41..9646243a9b 100644
> > --- a/block/qcow2-threads.c
> > +++ b/block/qcow2-threads.c
> > @@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
> >   }
> >   
> >   static int coroutine_fn
> > -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
> > -  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc 
> > func)
> > +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
> > +uint64_t guest_offset, void *buf, size_t len,
> > +Qcow2EncDecFunc func)
> >   {
> >   BDRVQcow2State *s = bs->opaque;
> > +
> >   Qcow2EncDecData arg = {
> >  

Re: [Qemu-devel] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

[Vladimir Sementsov-Ogievskiy]

13.09.2019 18:28, Maxim Levitsky wrote:
> Signed-off-by: Maxim Levitsky 
> ---
>   tests/qemu-iotests/263 | 91 ++
>   tests/qemu-iotests/263.out | 40 +
>   tests/qemu-iotests/group   |  2 +
>   3 files changed, 133 insertions(+)
>   create mode 100755 tests/qemu-iotests/263
>   create mode 100644 tests/qemu-iotests/263.out
> 
> diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
> new file mode 100755
> index 00..d2c030fae9
> --- /dev/null
> +++ b/tests/qemu-iotests/263

[..]

> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -274,5 +274,7 @@
>   257 rw
>   258 rw quick
>   262 rw quick migration
> +263 rw quick
>   265 rw auto quick
>   266 rw quick
> +>>> patched
> 

last line is a mistake.

also, test failed for me:

263  fail   [19:21:35] [19:21:35]output mismatch 
(see 263.out.bad)
--- /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out2019-09-13 
19:11:45.464727427 +0300
+++ /work/src/qemu/maxim-luks/tests/qemu-iotests/263.out.bad2019-09-13 
19:21:35.535381253 +0300
@@ -2,6 +2,7 @@

  testing LUKS qcow2 encryption

+qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this build: 
Function not implemented
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10
  == reading the whole image ==
  read 1048576/1048576 bytes at offset 0
Failures: 263
Failed 1 of 1 tests


and if reconfigure with
  --enable-gnutls --enable-gcrypt --enable-vhost-crypto --enable-crypto-afalg
(don't know which one is actually needed)
it works..

so, we at least should skip the test if it's unsupported


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

[Vladimir Sementsov-Ogievskiy]

13.09.2019 18:28, Maxim Levitsky wrote:
> Change do_perform_cow_encrypt and its callee qcow2_co_encrypt
> to just receive full host and guest offsets and in pariticular
> remove the offset_in_cluster parameter of do_perform_cow_encrypt,
> since it is misleading, because that offset can be larger than
> cluster size currently.
> 
> Also document the qcow2_co_encrypt arguments to prevent
> that bug from happening again
> 
> Signed-off-by: Maxim Levitsky 
> ---
>   block/qcow2-cluster.c | 30 -
>   block/qcow2-threads.c | 62 ++-
>   block/qcow2.c |  5 ++--
>   block/qcow2.h |  8 +++---
>   4 files changed, 73 insertions(+), 32 deletions(-)
> 
> diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> index bfeb0241d7..e87a4637fd 100644
> --- a/block/qcow2-cluster.c
> +++ b/block/qcow2-cluster.c
> @@ -463,21 +463,21 @@ static int coroutine_fn 
> do_perform_cow_read(BlockDriverState *bs,
>   }
>   
>   static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
> -uint64_t src_cluster_offset,
> -uint64_t cluster_offset,
> -unsigned offset_in_cluster,
> +uint64_t guest_offset,
> +uint64_t host_offset,
>   uint8_t *buffer,
>   unsigned bytes)
>   {
>   if (bytes && bs->encrypted) {
>   BDRVQcow2State *s = bs->opaque;
> -assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
> -assert((bytes & ~BDRV_SECTOR_MASK) == 0);
> +
> +assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
> +assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
> +assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
>   assert(s->crypto);
> -if (qcow2_co_encrypt(bs,
> -start_of_cluster(s, cluster_offset + offset_in_cluster),
> -src_cluster_offset + offset_in_cluster,
> -buffer, bytes) < 0) {
> +
> +if (qcow2_co_encrypt(bs, host_offset, guest_offset,
> +buffer, bytes) < 0) {

strange alignment of second line of the condition.. [1]

>   return false;
>   }
>   }
> @@ -891,11 +891,15 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta 
> *m)
>   
>   /* Encrypt the data if necessary before writing it */
>   if (bs->encrypted) {
> -if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> -start->offset, start_buffer,
> +if (!do_perform_cow_encrypt(bs,
> +m->offset + start->offset,
> +m->alloc_offset + start->offset,
> +start_buffer,
>   start->nb_bytes) ||
> -!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
> -end->offset, end_buffer, end->nb_bytes)) 
> {
> +!do_perform_cow_encrypt(bs,
> +m->offset + end->offset,
> +m->alloc_offset + end->offset,
> +end_buffer, end->nb_bytes)) {

Looking at this now, I think that do_perform_cow_encrypt can be dropped at all,
as it's now just an extra wrapper, sending same parameters to qcow2_co_encrypt.

I'll send a follow-up, if this patch goes as is.

>   ret = -EIO;
>   goto fail;
>   }
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 3b1e63fe41..9646243a9b 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
>   }
>   
>   static int coroutine_fn
> -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
> -  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc 
> func)
> +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
> +uint64_t guest_offset, void *buf, size_t len,
> +Qcow2EncDecFunc func)
>   {
>   BDRVQcow2State *s = bs->opaque;
> +
>   Qcow2EncDecData arg = {
>   .block = s->crypto,
> -.offset = s->crypt_physical_offset ?
> -  file_cluster_offset + offset_into_cluster(s, offset) :
> -  offset,
> +.offset = s->crypt_physical_offset ? host_offset : guest_offset,
>   .buf = buf,
>   .len = len,
>   .func = func,
> @@ -251,18 +251,54 @@ qcow2_co_encdec(BlockDriverState *bs, uint64_t 
> file_cluster_offset,
>   return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
>   }
>   
> +
> +/*
> + * qcow2_co_encrypt()
> + *
> + * Encrypts one or

[Qemu-devel] [PULL 10/12] aspeed/scu: Introduce a aspeed_scu_get_apb_freq() routine

[Peter Maydell]

From: Cédric Le Goater 

The APB frequency can be calculated directly when needed from the
HPLL_PARAM and CLK_SEL register values. This removes useless state in
the model.

Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-11-...@kaod.org
Reviewed-by: Peter Maydell 
Signed-off-by: Peter Maydell 
---
 include/hw/misc/aspeed_scu.h |  8 +++-
 hw/misc/aspeed_scu.c | 25 +
 hw/timer/aspeed_timer.c  |  3 ++-
 3 files changed, 14 insertions(+), 22 deletions(-)

diff --git a/include/hw/misc/aspeed_scu.h b/include/hw/misc/aspeed_scu.h
index 90dd4dadede..239e94fe2c4 100644
--- a/include/hw/misc/aspeed_scu.h
+++ b/include/hw/misc/aspeed_scu.h
@@ -32,10 +32,6 @@ typedef struct AspeedSCUState {
 uint32_t hw_strap1;
 uint32_t hw_strap2;
 uint32_t hw_prot_key;
-
-uint32_t clkin;
-uint32_t hpll;
-uint32_t apb_freq;
 } AspeedSCUState;
 
 #define AST2400_A0_SILICON_REV   0x02000303U
@@ -56,12 +52,14 @@ typedef struct  AspeedSCUClass {
 SysBusDeviceClass parent_class;
 
 const uint32_t *resets;
-uint32_t (*calc_hpll)(AspeedSCUState *s);
+uint32_t (*calc_hpll)(AspeedSCUState *s, uint32_t hpll_reg);
 uint32_t apb_divider;
 }  AspeedSCUClass;
 
 #define ASPEED_SCU_PROT_KEY  0x1688A8A8
 
+uint32_t aspeed_scu_get_apb_freq(AspeedSCUState *s);
+
 /*
  * Extracted from Aspeed SDK v00.03.21. Fixes and extra definitions
  * were added.
diff --git a/hw/misc/aspeed_scu.c b/hw/misc/aspeed_scu.c
index d284458b9b3..620b25c2047 100644
--- a/hw/misc/aspeed_scu.c
+++ b/hw/misc/aspeed_scu.c
@@ -164,11 +164,12 @@ static uint32_t aspeed_scu_get_random(void)
 return num;
 }
 
-static void aspeed_scu_set_apb_freq(AspeedSCUState *s)
+uint32_t aspeed_scu_get_apb_freq(AspeedSCUState *s)
 {
 AspeedSCUClass *asc = ASPEED_SCU_GET_CLASS(s);
+uint32_t hpll = asc->calc_hpll(s, s->regs[HPLL_PARAM]);
 
-s->apb_freq = s->hpll / (SCU_CLK_GET_PCLK_DIV(s->regs[CLK_SEL]) + 1)
+return hpll / (SCU_CLK_GET_PCLK_DIV(s->regs[CLK_SEL]) + 1)
 / asc->apb_divider;
 }
 
@@ -228,7 +229,6 @@ static void aspeed_scu_write(void *opaque, hwaddr offset, 
uint64_t data,
 return;
 case CLK_SEL:
 s->regs[reg] = data;
-aspeed_scu_set_apb_freq(s);
 break;
 case HW_STRAP1:
 if (ASPEED_IS_AST2500(s->regs[SILICON_REV])) {
@@ -290,11 +290,11 @@ static const uint32_t hpll_ast2400_freqs[][4] = {
 { 400, 375, 350, 425 }, /* 25MHz */
 };
 
-static uint32_t aspeed_2400_scu_calc_hpll(AspeedSCUState *s)
+static uint32_t aspeed_2400_scu_calc_hpll(AspeedSCUState *s, uint32_t hpll_reg)
 {
-uint32_t hpll_reg = s->regs[HPLL_PARAM];
 uint8_t freq_select;
 bool clk_25m_in;
+uint32_t clkin = aspeed_scu_get_clkin(s);
 
 if (hpll_reg & SCU_AST2400_H_PLL_OFF) {
 return 0;
@@ -311,7 +311,7 @@ static uint32_t aspeed_2400_scu_calc_hpll(AspeedSCUState *s)
 multiplier = (2 - od) * ((n + 2) / (d + 1));
 }
 
-return s->clkin * multiplier;
+return clkin * multiplier;
 }
 
 /* HW strapping */
@@ -321,10 +321,10 @@ static uint32_t aspeed_2400_scu_calc_hpll(AspeedSCUState 
*s)
 return hpll_ast2400_freqs[clk_25m_in][freq_select] * 100;
 }
 
-static uint32_t aspeed_2500_scu_calc_hpll(AspeedSCUState *s)
+static uint32_t aspeed_2500_scu_calc_hpll(AspeedSCUState *s, uint32_t hpll_reg)
 {
-uint32_t hpll_reg   = s->regs[HPLL_PARAM];
 uint32_t multiplier = 1;
+uint32_t clkin = aspeed_scu_get_clkin(s);
 
 if (hpll_reg & SCU_H_PLL_OFF) {
 return 0;
@@ -338,7 +338,7 @@ static uint32_t aspeed_2500_scu_calc_hpll(AspeedSCUState *s)
 multiplier = ((m + 1) / (n + 1)) / (p + 1);
 }
 
-return s->clkin * multiplier;
+return clkin * multiplier;
 }
 
 static void aspeed_scu_reset(DeviceState *dev)
@@ -351,13 +351,6 @@ static void aspeed_scu_reset(DeviceState *dev)
 s->regs[HW_STRAP1] = s->hw_strap1;
 s->regs[HW_STRAP2] = s->hw_strap2;
 s->regs[PROT_KEY] = s->hw_prot_key;
-
-/*
- * All registers are set. Now compute the frequencies of the main clocks
- */
-s->clkin = aspeed_scu_get_clkin(s);
-s->hpll = asc->calc_hpll(s);
-aspeed_scu_set_apb_freq(s);
 }
 
 static uint32_t aspeed_silicon_revs[] = {
diff --git a/hw/timer/aspeed_timer.c b/hw/timer/aspeed_timer.c
index 59c2bbeee60..2bda826882d 100644
--- a/hw/timer/aspeed_timer.c
+++ b/hw/timer/aspeed_timer.c
@@ -93,7 +93,8 @@ static inline uint32_t calculate_rate(struct AspeedTimer *t)
 {
 AspeedTimerCtrlState *s = timer_to_ctrl(t);
 
-return timer_external_clock(t) ? TIMER_CLOCK_EXT_HZ : s->scu->apb_freq;
+return timer_external_clock(t) ? TIMER_CLOCK_EXT_HZ :
+aspeed_scu_get_apb_freq(s->scu);
 }
 
 static inline uint32_t calculate_ticks(struct AspeedTimer *t, uint64_t now_ns)
-- 
2.20.1

[Qemu-devel] [PULL 12/12] qemu-ga: Convert invocation documentation to rST

[Peter Maydell]

The qemu-ga documentation is currently in qemu-ga.texi in
Texinfo format, which we present to the user as:
 * a qemu-ga manpage
 * a section of the main qemu-doc HTML documentation

Convert the documentation to rST format, and present it to
the user as:
 * a qemu-ga manpage
 * part of the interop/ Sphinx manual

Signed-off-by: Peter Maydell 
Reviewed-by: Michael Roth 
Tested-by: Michael Roth 
Message-id: 20190905131040.8350-1-peter.mayd...@linaro.org
---
 Makefile |  24 ---
 MAINTAINERS  |   2 +-
 docs/conf.py |  18 ++---
 docs/interop/conf.py |   7 ++
 docs/interop/index.rst   |   1 +
 docs/interop/qemu-ga.rst | 133 +
 qemu-doc.texi|   5 --
 qemu-ga.texi | 137 ---
 8 files changed, 166 insertions(+), 161 deletions(-)
 create mode 100644 docs/interop/qemu-ga.rst
 delete mode 100644 qemu-ga.texi

diff --git a/Makefile b/Makefile
index b3528617e48..111082ce545 100644
--- a/Makefile
+++ b/Makefile
@@ -325,7 +325,7 @@ endif
 endif
 
 ifdef BUILD_DOCS
-DOCS=qemu-doc.html qemu-doc.txt qemu.1 qemu-img.1 qemu-nbd.8 qemu-ga.8
+DOCS=qemu-doc.html qemu-doc.txt qemu.1 qemu-img.1 qemu-nbd.8 
docs/interop/qemu-ga.8
 DOCS+=docs/interop/qemu-qmp-ref.html docs/interop/qemu-qmp-ref.txt 
docs/interop/qemu-qmp-ref.7
 DOCS+=docs/interop/qemu-ga-ref.html docs/interop/qemu-ga-ref.txt 
docs/interop/qemu-ga-ref.7
 DOCS+=docs/qemu-block-drivers.7
@@ -783,10 +783,11 @@ DESCS=
 endif
 
 # Note that we manually filter-out the non-Sphinx documentation which
-# is currently built into the docs/interop directory in the build tree.
+# is currently built into the docs/interop directory in the build tree,
+# and also any sphinx-built manpages.
 define install-manual =
 for d in $$(cd $(MANUAL_BUILDDIR) && find $1 -type d); do $(INSTALL_DIR) 
"$(DESTDIR)$(qemu_docdir)/$$d"; done
-for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f -a '!' '(' -name 
'qemu-*-qapi.*' -o -name 'qemu-*-ref.*' ')' ); do $(INSTALL_DATA) 
"$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
+for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f -a '!' '(' -name 
'*.[0-9]' -o -name 'qemu-*-qapi.*' -o -name 'qemu-*-ref.*' ')' ); do 
$(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
 endef
 
 # Note that we deliberately do not install the "devel" manual: it is
@@ -818,7 +819,7 @@ ifdef CONFIG_TRACE_SYSTEMTAP
$(INSTALL_DATA) scripts/qemu-trace-stap.1 "$(DESTDIR)$(mandir)/man1"
 endif
 ifneq (,$(findstring qemu-ga,$(TOOLS)))
-   $(INSTALL_DATA) qemu-ga.8 "$(DESTDIR)$(mandir)/man8"
+   $(INSTALL_DATA) docs/interop/qemu-ga.8 "$(DESTDIR)$(mandir)/man8"
$(INSTALL_DATA) docs/interop/qemu-ga-ref.html "$(DESTDIR)$(qemu_docdir)"
$(INSTALL_DATA) docs/interop/qemu-ga-ref.txt "$(DESTDIR)$(qemu_docdir)"
$(INSTALL_DATA) docs/interop/qemu-ga-ref.7 "$(DESTDIR)$(mandir)/man7"
@@ -977,18 +978,22 @@ docs/version.texi: $(SRC_PATH)/VERSION config-host.mak
 sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html 
$(MANUAL_BUILDDIR)/interop/index.html $(MANUAL_BUILDDIR)/specs/index.html
 
 # Canned command to build a single manual
-build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -W -n -b html 
-D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 
$(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
+# Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
+build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build 
$(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" 
-d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 
,"SPHINX","$(MANUAL_BUILDDIR)/$1")
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) 
$(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 
 $(MANUAL_BUILDDIR)/devel/index.html: $(call manual-deps,devel)
-   $(call build-manual,devel)
+   $(call build-manual,devel,html)
 
 $(MANUAL_BUILDDIR)/interop/index.html: $(call manual-deps,interop)
-   $(call build-manual,interop)
+   $(call build-manual,interop,html)
 
 $(MANUAL_BUILDDIR)/specs/index.html: $(call manual-deps,specs)
-   $(call build-manual,specs)
+   $(call build-manual,specs,html)
+
+$(MANUAL_BUILDDIR)/interop/qemu-ga.8: $(call manual-deps,interop)
+   $(call build-manual,interop,man)
 
 qemu-options.texi: $(SRC_PATH)/qemu-options.hx $(SRC_PATH)/scripts/hxtool
$(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > 
$@,"GEN","$@")
@@ -1013,7 +1018,6 @@ qemu.1: qemu-option-trace.texi
 qemu-img.1: qemu-img.texi qemu-option-trace.texi qemu-img-cmds.texi
 fsdev/virtfs-proxy-helper.1: fsdev/virtfs-proxy-helper.texi
 qemu-nbd.8: qemu-nbd.texi qemu-option-trace.texi
-qemu-ga.8: qemu-ga.texi
 docs/qemu-block-drivers.7: docs/qemu-block-drivers.texi
 docs/qemu-cpu-models.7: docs/qemu-cpu-models.tex

[Qemu-devel] [PULL 07/12] aspeed/smc: Inject errors in DMA checksum

[Peter Maydell]

From: Cédric Le Goater 

Emulate read errors in the DMA Checksum Register for high frequencies
and optimistic settings of the Read Timing Compensation Register. This
will help in tuning the SPI timing calibration algorithm. Errors are
only injected when the property "inject_failure" is set to true as
suggested by Philippe.

The values below are those to expect from the first flash device of
the FMC controller of a palmetto-bmc machine.

Cc: Philippe Mathieu-Daudé 
Signed-off-by: Cédric Le Goater 
Reviewed-by: Joel Stanley 
Message-id: 20190904070506.1052-8-...@kaod.org
Signed-off-by: Peter Maydell 
---
 include/hw/ssi/aspeed_smc.h |  1 +
 hw/ssi/aspeed_smc.c | 36 
 2 files changed, 37 insertions(+)

diff --git a/include/hw/ssi/aspeed_smc.h b/include/hw/ssi/aspeed_smc.h
index 32ce6916f6c..5176ff6bf95 100644
--- a/include/hw/ssi/aspeed_smc.h
+++ b/include/hw/ssi/aspeed_smc.h
@@ -88,6 +88,7 @@ typedef struct AspeedSMCState {
 
 uint32_t num_cs;
 qemu_irq *cs_lines;
+bool inject_failure;
 
 SSIBus *spi;
 
diff --git a/hw/ssi/aspeed_smc.c b/hw/ssi/aspeed_smc.c
index 7a0cd7607fd..5c3436db5e8 100644
--- a/hw/ssi/aspeed_smc.c
+++ b/hw/ssi/aspeed_smc.c
@@ -869,6 +869,36 @@ static void aspeed_smc_dma_calibration(AspeedSMCState *s)
 s->regs[s->r_ctrl0 + cs] |= CE_CTRL_CLOCK_FREQ(hclk_div);
 }
 
+/*
+ * Emulate read errors in the DMA Checksum Register for high
+ * frequencies and optimistic settings of the Read Timing Compensation
+ * Register. This will help in tuning the SPI timing calibration
+ * algorithm.
+ */
+static bool aspeed_smc_inject_read_failure(AspeedSMCState *s)
+{
+uint8_t delay =
+(s->regs[R_DMA_CTRL] >> DMA_CTRL_DELAY_SHIFT) & DMA_CTRL_DELAY_MASK;
+uint8_t hclk_mask =
+(s->regs[R_DMA_CTRL] >> DMA_CTRL_FREQ_SHIFT) & DMA_CTRL_FREQ_MASK;
+
+/*
+ * Typical values of a palmetto-bmc machine.
+ */
+switch (aspeed_smc_hclk_divisor(hclk_mask)) {
+case 4 ... 16:
+return false;
+case 3: /* at least one HCLK cycle delay */
+return (delay & 0x7) < 1;
+case 2: /* at least two HCLK cycle delay */
+return (delay & 0x7) < 2;
+case 1: /* (> 100MHz) is above the max freq of the controller */
+return true;
+default:
+g_assert_not_reached();
+}
+}
+
 /*
  * Accumulate the result of the reads to provide a checksum that will
  * be used to validate the read timing settings.
@@ -905,6 +935,11 @@ static void aspeed_smc_dma_checksum(AspeedSMCState *s)
 s->regs[R_DMA_FLASH_ADDR] += 4;
 s->regs[R_DMA_LEN] -= 4;
 }
+
+if (s->inject_failure && aspeed_smc_inject_read_failure(s)) {
+s->regs[R_DMA_CHECKSUM] = 0xbadc0de;
+}
+
 }
 
 static void aspeed_smc_dma_rw(AspeedSMCState *s)
@@ -1185,6 +1220,7 @@ static const VMStateDescription vmstate_aspeed_smc = {
 
 static Property aspeed_smc_properties[] = {
 DEFINE_PROP_UINT32("num-cs", AspeedSMCState, num_cs, 1),
+DEFINE_PROP_BOOL("inject-failure", AspeedSMCState, inject_failure, false),
 DEFINE_PROP_UINT64("sdram-base", AspeedSMCState, sdram_base, 0),
 DEFINE_PROP_LINK("dram", AspeedSMCState, dram_mr,
  TYPE_MEMORY_REGION, MemoryRegion *),
-- 
2.20.1

[Qemu-devel] [PULL 05/12] aspeed/smc: Add support for DMAs

[Peter Maydell]

From: Cédric Le Goater 

The FMC controller on the Aspeed SoCs support DMA to access the flash
modules. It can operate in a normal mode, to copy to or from the flash
module mapping window, or in a checksum calculation mode, to evaluate
the best clock settings for reads.

The model introduces two custom address spaces for DMAs: one for the
AHB window of the FMC flash devices and one for the DRAM. The latter
is populated using a "dram" link set from the machine with the RAM
container region.

Signed-off-by: Cédric Le Goater 
Acked-by: Joel Stanley 
Message-id: 20190904070506.1052-6-...@kaod.org
Reviewed-by: Peter Maydell 
Signed-off-by: Peter Maydell 
---
 include/hw/ssi/aspeed_smc.h |   6 +
 hw/arm/aspeed.c |   2 +
 hw/arm/aspeed_soc.c |   2 +
 hw/ssi/aspeed_smc.c | 222 +++-
 4 files changed, 226 insertions(+), 6 deletions(-)

diff --git a/include/hw/ssi/aspeed_smc.h b/include/hw/ssi/aspeed_smc.h
index aa07dac4fe3..32ce6916f6c 100644
--- a/include/hw/ssi/aspeed_smc.h
+++ b/include/hw/ssi/aspeed_smc.h
@@ -46,6 +46,8 @@ typedef struct AspeedSMCController {
 hwaddr flash_window_base;
 uint32_t flash_window_size;
 bool has_dma;
+hwaddr dma_flash_mask;
+hwaddr dma_dram_mask;
 uint32_t nregs;
 } AspeedSMCController;
 
@@ -101,6 +103,10 @@ typedef struct AspeedSMCState {
 /* for DMA support */
 uint64_t sdram_base;
 
+AddressSpace flash_as;
+MemoryRegion *dram_mr;
+AddressSpace dram_as;
+
 AspeedSMCFlash *flashes;
 
 uint8_t snoop_index;
diff --git a/hw/arm/aspeed.c b/hw/arm/aspeed.c
index 13e208c78cc..aa72be309da 100644
--- a/hw/arm/aspeed.c
+++ b/hw/arm/aspeed.c
@@ -190,6 +190,8 @@ static void aspeed_board_init(MachineState *machine,
 &error_abort);
 object_property_set_int(OBJECT(&bmc->soc), machine->smp.cpus, "num-cpus",
 &error_abort);
+object_property_set_link(OBJECT(&bmc->soc), OBJECT(&bmc->ram_container),
+ "dram", &error_abort);
 if (machine->kernel_filename) {
 /*
  * When booting with a -kernel command line there is no u-boot
diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
index 25dbc409d35..da508c99c33 100644
--- a/hw/arm/aspeed_soc.c
+++ b/hw/arm/aspeed_soc.c
@@ -191,6 +191,8 @@ static void aspeed_soc_init(Object *obj)
   typename);
 object_property_add_alias(obj, "num-cs", OBJECT(&s->fmc), "num-cs",
   &error_abort);
+object_property_add_alias(obj, "dram", OBJECT(&s->fmc), "dram",
+  &error_abort);
 
 for (i = 0; i < sc->info->spis_num; i++) {
 snprintf(typename, sizeof(typename), "aspeed.spi%d-%s", i + 1, 
socname);
diff --git a/hw/ssi/aspeed_smc.c b/hw/ssi/aspeed_smc.c
index f4f7c181830..c1a45c10dc1 100644
--- a/hw/ssi/aspeed_smc.c
+++ b/hw/ssi/aspeed_smc.c
@@ -28,6 +28,8 @@
 #include "qemu/log.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "exec/address-spaces.h"
 
 #include "hw/irq.h"
 #include "hw/qdev-properties.h"
@@ -112,8 +114,8 @@
 #define   DMA_CTRL_FREQ_SHIFT   4
 #define   DMA_CTRL_MODE (1 << 3)
 #define   DMA_CTRL_CKSUM(1 << 2)
-#define   DMA_CTRL_DIR  (1 << 1)
-#define   DMA_CTRL_EN   (1 << 0)
+#define   DMA_CTRL_WRITE(1 << 1)
+#define   DMA_CTRL_ENABLE   (1 << 0)
 
 /* DMA Flash Side Address */
 #define R_DMA_FLASH_ADDR  (0x84 / 4)
@@ -145,6 +147,24 @@
 #define ASPEED_SOC_SPI_FLASH_BASE   0x3000
 #define ASPEED_SOC_SPI2_FLASH_BASE  0x3800
 
+/*
+ * DMA DRAM addresses should be 4 bytes aligned and the valid address
+ * range is 0x4000 - 0x5FFF (AST2400)
+ *  0x8000 - 0xBFFF (AST2500)
+ *
+ * DMA flash addresses should be 4 bytes aligned and the valid address
+ * range is 0x2000 - 0x2FFF.
+ *
+ * DMA length is from 4 bytes to 32MB
+ *   0: 4 bytes
+ *   0x7F: 32M bytes
+ */
+#define DMA_DRAM_ADDR(s, val)   ((s)->sdram_base | \
+ ((val) & (s)->ctrl->dma_dram_mask))
+#define DMA_FLASH_ADDR(s, val)  ((s)->ctrl->flash_window_base | \
+((val) & (s)->ctrl->dma_flash_mask))
+#define DMA_LENGTH(val) ((val) & 0x01FC)
+
 /* Flash opcodes. */
 #define SPI_OP_READ   0x03/* Read data bytes (low frequency) */
 
@@ -214,6 +234,8 @@ static const AspeedSMCController controllers[] = {
 .flash_window_base = ASPEED_SOC_FMC_FLASH_BASE,
 .flash_window_size = 0x1000,
 .has_dma   = true,
+.dma_flash_mask= 0x0FFC,
+.dma_dram_mask = 0x1FFC,
 .nregs = ASPEED_SMC_R_MAX,
 }, {
 .name  = "aspeed.spi1-ast2400",
@@ -240,6 +262,8 @@ static const AspeedSMCController controllers[] = {
 .flash_window_base = ASPEED_SOC_FMC_FLASH_BASE,
 .flash_window_size =

Re: [Qemu-devel] [Bug 1843852] Re: QEMU does not express a dependency on perl-Test-Harness

[John Snow]

On 9/13/19 11:06 AM, Paolo Bonzini wrote:
> On 13/09/19 16:56, John Snow wrote:
>>
>>
>> On 9/13/19 4:33 AM, Alex Bennée wrote:
>>> Given we require python perhaps the simplest solution would be to re-
>>> write the tap-driver as a python script rather than adding another
>>> configure check?
>>
>> Seems a shame to need to after Paolo *just* introduced this perl script
>> late last year in 9df43317b82; it looks well-written as far as perl goes.
> 
> Also because I didn't write it most of it (rather, it comes from
> Automake).  The new dependency was even documented in the release notes:
> 
>   Running the QEMU testsuite now requires the Perl Test::Harness module.
>   Most Linux and BSD distributions however install it by default
>   together with Perl.
> 

Yeah, let's just add a warning(?) to the configure check that you'll be
unable to run `make check` if you're missing this dependency. Easier
than re-writing.

--js

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1843852

Title:
  QEMU does not express a dependency on perl-Test-Harness

Status in QEMU:
  New

Bug description:
  This is a minor thing; in Fedora you can install most of the developer
  dependencies by issuing something like `dnf builddep qemu-kvm` and
  this takes care of just about everything such that you can run
  ./configure and make.

  For "make check" though, configure doesn't catch that you'll need
  perl-Test-Harness; so it fails halfway through the check routine, and
  you'll see this:

  ```
  Can't locate TAP/Parser.pm in @INC (you may need to install the TAP::Parser 
module) (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 
/usr/share/perl5) at ./scripts/tap-driver.pl line 30.
  BEGIN failed--compilation aborted at ./scripts/tap-driver.pl line 30.
  make: *** [/home/jhuston/src/qemu/tests/Makefile.include:905: check-unit] 
Error 2
  ```

  I'm not sure how we should express this dependency; it shouldn't be a
  requirement for building, but it IS a dependency for testing. We
  probably ought not let users skip the qapi tests just because they
  don't have the perl requirement met.

  (And, separately, the Fedora package should list this as a builddep,
  but that's not an issue for here.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1843852/+subscriptions

[Qemu-devel] [PULL 06/12] aspeed/smc: Add DMA calibration settings

[Peter Maydell]

From: Cédric Le Goater 

When doing calibration, the SPI clock rate in the CE0 Control Register
and the read delay cycles in the Read Timing Compensation Register are
set using bit[11:4] of the DMA Control Register.

Signed-off-by: Cédric Le Goater 
Acked-by: Joel Stanley 
Reviewed-by: Peter Maydell 
Message-id: 20190904070506.1052-7-...@kaod.org
Signed-off-by: Peter Maydell 
---
 hw/ssi/aspeed_smc.c | 64 -
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/hw/ssi/aspeed_smc.c b/hw/ssi/aspeed_smc.c
index c1a45c10dc1..7a0cd7607fd 100644
--- a/hw/ssi/aspeed_smc.c
+++ b/hw/ssi/aspeed_smc.c
@@ -77,6 +77,10 @@
 #define   CTRL_CMD_MASK0xff
 #define   CTRL_DUMMY_HIGH_SHIFT14
 #define   CTRL_AST2400_SPI_4BYTE   (1 << 13)
+#define CE_CTRL_CLOCK_FREQ_SHIFT   8
+#define CE_CTRL_CLOCK_FREQ_MASK0xf
+#define CE_CTRL_CLOCK_FREQ(div) \
+(((div) & CE_CTRL_CLOCK_FREQ_MASK) << CE_CTRL_CLOCK_FREQ_SHIFT)
 #define   CTRL_DUMMY_LOW_SHIFT 6 /* 2 bits [7:6] */
 #define   CTRL_CE_STOP_ACTIVE  (1 << 2)
 #define   CTRL_CMD_MODE_MASK   0x3
@@ -112,7 +116,7 @@
 #define   DMA_CTRL_DELAY_SHIFT  8
 #define   DMA_CTRL_FREQ_MASK0xf
 #define   DMA_CTRL_FREQ_SHIFT   4
-#define   DMA_CTRL_MODE (1 << 3)
+#define   DMA_CTRL_CALIB(1 << 3)
 #define   DMA_CTRL_CKSUM(1 << 2)
 #define   DMA_CTRL_WRITE(1 << 1)
 #define   DMA_CTRL_ENABLE   (1 << 0)
@@ -811,6 +815,60 @@ static uint64_t aspeed_smc_read(void *opaque, hwaddr addr, 
unsigned int size)
 }
 }
 
+static uint8_t aspeed_smc_hclk_divisor(uint8_t hclk_mask)
+{
+/* HCLK/1 .. HCLK/16 */
+const uint8_t hclk_divisors[] = {
+15, 7, 14, 6, 13, 5, 12, 4, 11, 3, 10, 2, 9, 1, 8, 0
+};
+int i;
+
+for (i = 0; i < ARRAY_SIZE(hclk_divisors); i++) {
+if (hclk_mask == hclk_divisors[i]) {
+return i + 1;
+}
+}
+
+qemu_log_mask(LOG_GUEST_ERROR, "invalid HCLK mask %x", hclk_mask);
+return 0;
+}
+
+/*
+ * When doing calibration, the SPI clock rate in the CE0 Control
+ * Register and the read delay cycles in the Read Timing Compensation
+ * Register are set using bit[11:4] of the DMA Control Register.
+ */
+static void aspeed_smc_dma_calibration(AspeedSMCState *s)
+{
+uint8_t delay =
+(s->regs[R_DMA_CTRL] >> DMA_CTRL_DELAY_SHIFT) & DMA_CTRL_DELAY_MASK;
+uint8_t hclk_mask =
+(s->regs[R_DMA_CTRL] >> DMA_CTRL_FREQ_SHIFT) & DMA_CTRL_FREQ_MASK;
+uint8_t hclk_div = aspeed_smc_hclk_divisor(hclk_mask);
+uint32_t hclk_shift = (hclk_div - 1) << 2;
+uint8_t cs;
+
+/*
+ * The Read Timing Compensation Register values apply to all CS on
+ * the SPI bus and only HCLK/1 - HCLK/5 can have tunable delays
+ */
+if (hclk_div && hclk_div < 6) {
+s->regs[s->r_timings] &= ~(0xf << hclk_shift);
+s->regs[s->r_timings] |= delay << hclk_shift;
+}
+
+/*
+ * TODO: compute the CS from the DMA address and the segment
+ * registers. This is not really a problem for now because the
+ * Timing Register values apply to all CS and software uses CS0 to
+ * do calibration.
+ */
+cs = 0;
+s->regs[s->r_ctrl0 + cs] &=
+~(CE_CTRL_CLOCK_FREQ_MASK << CE_CTRL_CLOCK_FREQ_SHIFT);
+s->regs[s->r_ctrl0 + cs] |= CE_CTRL_CLOCK_FREQ(hclk_div);
+}
+
 /*
  * Accumulate the result of the reads to provide a checksum that will
  * be used to validate the read timing settings.
@@ -826,6 +884,10 @@ static void aspeed_smc_dma_checksum(AspeedSMCState *s)
 return;
 }
 
+if (s->regs[R_DMA_CTRL] & DMA_CTRL_CALIB) {
+aspeed_smc_dma_calibration(s);
+}
+
 while (s->regs[R_DMA_LEN]) {
 data = address_space_ldl_le(&s->flash_as, s->regs[R_DMA_FLASH_ADDR],
 MEMTXATTRS_UNSPECIFIED, &result);
-- 
2.20.1

[Qemu-devel] [PULL 11/12] atomic_template: fix indentation in GEN_ATOMIC_HELPER

[Peter Maydell]

From: "Emilio G. Cota" 

Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Reviewed-by: Richard Henderson 
Signed-off-by: Alex Bennée 
Signed-off-by: Peter Maydell 
---
 accel/tcg/atomic_template.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/accel/tcg/atomic_template.h b/accel/tcg/atomic_template.h
index df9c8388178..287433d809b 100644
--- a/accel/tcg/atomic_template.h
+++ b/accel/tcg/atomic_template.h
@@ -149,7 +149,7 @@ ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong 
addr,
 
 #define GEN_ATOMIC_HELPER(X)\
 ABI_TYPE ATOMIC_NAME(X)(CPUArchState *env, target_ulong addr,   \
- ABI_TYPE val EXTRA_ARGS)   \
+ABI_TYPE val EXTRA_ARGS)\
 {   \
 ATOMIC_MMU_DECLS;   \
 DATA_TYPE *haddr = ATOMIC_MMU_LOOKUP;   \
-- 
2.20.1

[Qemu-devel] [PULL 02/12] aspeed: add a GPIO controller to the SoC

[Peter Maydell]

From: Rashmica Gupta 

Signed-off-by: Rashmica Gupta 
Reviewed-by: Cédric Le Goater 
Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-3-...@kaod.org
Signed-off-by: Peter Maydell 
---
 include/hw/arm/aspeed_soc.h |  3 +++
 hw/arm/aspeed_soc.c | 17 +
 2 files changed, 20 insertions(+)

diff --git a/include/hw/arm/aspeed_soc.h b/include/hw/arm/aspeed_soc.h
index 976fd6be93f..a56effebc16 100644
--- a/include/hw/arm/aspeed_soc.h
+++ b/include/hw/arm/aspeed_soc.h
@@ -23,6 +23,7 @@
 #include "hw/watchdog/wdt_aspeed.h"
 #include "hw/net/ftgmac100.h"
 #include "target/arm/cpu.h"
+#include "hw/gpio/aspeed_gpio.h"
 
 #define ASPEED_SPIS_NUM  2
 #define ASPEED_WDTS_NUM  3
@@ -48,6 +49,7 @@ typedef struct AspeedSoCState {
 AspeedSDMCState sdmc;
 AspeedWDTState wdt[ASPEED_WDTS_NUM];
 FTGMAC100State ftgmac100[ASPEED_MACS_NUM];
+AspeedGPIOState gpio;
 } AspeedSoCState;
 
 #define TYPE_ASPEED_SOC "aspeed-soc"
@@ -61,6 +63,7 @@ typedef struct AspeedSoCInfo {
 int spis_num;
 const char *fmc_typename;
 const char **spi_typename;
+const char *gpio_typename;
 int wdts_num;
 const int *irqmap;
 const hwaddr *memmap;
diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
index 9ee81048326..04480875d0d 100644
--- a/hw/arm/aspeed_soc.c
+++ b/hw/arm/aspeed_soc.c
@@ -125,6 +125,7 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .spis_num = 1,
 .fmc_typename = "aspeed.smc.fmc",
 .spi_typename = aspeed_soc_ast2400_typenames,
+.gpio_typename = "aspeed.gpio-ast2400",
 .wdts_num = 2,
 .irqmap   = aspeed_soc_ast2400_irqmap,
 .memmap   = aspeed_soc_ast2400_memmap,
@@ -137,6 +138,7 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .spis_num = 1,
 .fmc_typename = "aspeed.smc.fmc",
 .spi_typename = aspeed_soc_ast2400_typenames,
+.gpio_typename = "aspeed.gpio-ast2400",
 .wdts_num = 2,
 .irqmap   = aspeed_soc_ast2400_irqmap,
 .memmap   = aspeed_soc_ast2400_memmap,
@@ -149,6 +151,7 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .spis_num = 1,
 .fmc_typename = "aspeed.smc.fmc",
 .spi_typename = aspeed_soc_ast2400_typenames,
+.gpio_typename = "aspeed.gpio-ast2400",
 .wdts_num = 2,
 .irqmap   = aspeed_soc_ast2400_irqmap,
 .memmap   = aspeed_soc_ast2400_memmap,
@@ -161,6 +164,7 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .spis_num = 2,
 .fmc_typename = "aspeed.smc.ast2500-fmc",
 .spi_typename = aspeed_soc_ast2500_typenames,
+.gpio_typename = "aspeed.gpio-ast2500",
 .wdts_num = 3,
 .irqmap   = aspeed_soc_ast2500_irqmap,
 .memmap   = aspeed_soc_ast2500_memmap,
@@ -247,6 +251,9 @@ static void aspeed_soc_init(Object *obj)
 
 sysbus_init_child_obj(obj, "xdma", OBJECT(&s->xdma), sizeof(s->xdma),
   TYPE_ASPEED_XDMA);
+
+sysbus_init_child_obj(obj, "gpio", OBJECT(&s->gpio), sizeof(s->gpio),
+  sc->info->gpio_typename);
 }
 
 static void aspeed_soc_realize(DeviceState *dev, Error **errp)
@@ -426,6 +433,16 @@ static void aspeed_soc_realize(DeviceState *dev, Error 
**errp)
 sc->info->memmap[ASPEED_XDMA]);
 sysbus_connect_irq(SYS_BUS_DEVICE(&s->xdma), 0,
aspeed_soc_get_irq(s, ASPEED_XDMA));
+
+/* GPIO */
+object_property_set_bool(OBJECT(&s->gpio), true, "realized", &err);
+if (err) {
+error_propagate(errp, err);
+return;
+}
+sysbus_mmio_map(SYS_BUS_DEVICE(&s->gpio), 0, 
sc->info->memmap[ASPEED_GPIO]);
+sysbus_connect_irq(SYS_BUS_DEVICE(&s->gpio), 0,
+   aspeed_soc_get_irq(s, ASPEED_GPIO));
 }
 static Property aspeed_soc_properties[] = {
 DEFINE_PROP_UINT32("num-cpus", AspeedSoCState, num_cpus, 0),
-- 
2.20.1

[Qemu-devel] [PULL 09/12] aspeed/scu: Introduce per-SoC SCU types

[Peter Maydell]

From: Cédric Le Goater 

and use a class AspeedSCUClass to define each SoC characteristics.

Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-10-...@kaod.org
Reviewed-by: Peter Maydell 
Signed-off-by: Peter Maydell 
---
 include/hw/misc/aspeed_scu.h | 15 +++
 hw/arm/aspeed_soc.c  |  3 +-
 hw/misc/aspeed_scu.c | 83 
 3 files changed, 64 insertions(+), 37 deletions(-)

diff --git a/include/hw/misc/aspeed_scu.h b/include/hw/misc/aspeed_scu.h
index 38996adc59f..90dd4dadede 100644
--- a/include/hw/misc/aspeed_scu.h
+++ b/include/hw/misc/aspeed_scu.h
@@ -15,6 +15,8 @@
 
 #define TYPE_ASPEED_SCU "aspeed.scu"
 #define ASPEED_SCU(obj) OBJECT_CHECK(AspeedSCUState, (obj), TYPE_ASPEED_SCU)
+#define TYPE_ASPEED_2400_SCU TYPE_ASPEED_SCU "-ast2400"
+#define TYPE_ASPEED_2500_SCU TYPE_ASPEED_SCU "-ast2500"
 
 #define ASPEED_SCU_NR_REGS (0x1A8 >> 2)
 
@@ -45,6 +47,19 @@ typedef struct AspeedSCUState {
 
 extern bool is_supported_silicon_rev(uint32_t silicon_rev);
 
+#define ASPEED_SCU_CLASS(klass) \
+ OBJECT_CLASS_CHECK(AspeedSCUClass, (klass), TYPE_ASPEED_SCU)
+#define ASPEED_SCU_GET_CLASS(obj) \
+ OBJECT_GET_CLASS(AspeedSCUClass, (obj), TYPE_ASPEED_SCU)
+
+typedef struct  AspeedSCUClass {
+SysBusDeviceClass parent_class;
+
+const uint32_t *resets;
+uint32_t (*calc_hpll)(AspeedSCUState *s);
+uint32_t apb_divider;
+}  AspeedSCUClass;
+
 #define ASPEED_SCU_PROT_KEY  0x1688A8A8
 
 /*
diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
index da508c99c33..cf1d0cf921b 100644
--- a/hw/arm/aspeed_soc.c
+++ b/hw/arm/aspeed_soc.c
@@ -161,8 +161,9 @@ static void aspeed_soc_init(Object *obj)
 &error_abort, NULL);
 }
 
+snprintf(typename, sizeof(typename), "aspeed.scu-%s", socname);
 sysbus_init_child_obj(obj, "scu", OBJECT(&s->scu), sizeof(s->scu),
-  TYPE_ASPEED_SCU);
+  typename);
 qdev_prop_set_uint32(DEVICE(&s->scu), "silicon-rev",
  sc->info->silicon_rev);
 object_property_add_alias(obj, "hw-strap1", OBJECT(&s->scu),
diff --git a/hw/misc/aspeed_scu.c b/hw/misc/aspeed_scu.c
index 268cb24e565..d284458b9b3 100644
--- a/hw/misc/aspeed_scu.c
+++ b/hw/misc/aspeed_scu.c
@@ -166,23 +166,10 @@ static uint32_t aspeed_scu_get_random(void)
 
 static void aspeed_scu_set_apb_freq(AspeedSCUState *s)
 {
-uint32_t apb_divider;
-
-switch (s->silicon_rev) {
-case AST2400_A0_SILICON_REV:
-case AST2400_A1_SILICON_REV:
-apb_divider = 2;
-break;
-case AST2500_A0_SILICON_REV:
-case AST2500_A1_SILICON_REV:
-apb_divider = 4;
-break;
-default:
-g_assert_not_reached();
-}
+AspeedSCUClass *asc = ASPEED_SCU_GET_CLASS(s);
 
 s->apb_freq = s->hpll / (SCU_CLK_GET_PCLK_DIV(s->regs[CLK_SEL]) + 1)
-/ apb_divider;
+/ asc->apb_divider;
 }
 
 static uint64_t aspeed_scu_read(void *opaque, hwaddr offset, unsigned size)
@@ -303,7 +290,7 @@ static const uint32_t hpll_ast2400_freqs[][4] = {
 { 400, 375, 350, 425 }, /* 25MHz */
 };
 
-static uint32_t aspeed_scu_calc_hpll_ast2400(AspeedSCUState *s)
+static uint32_t aspeed_2400_scu_calc_hpll(AspeedSCUState *s)
 {
 uint32_t hpll_reg = s->regs[HPLL_PARAM];
 uint8_t freq_select;
@@ -334,7 +321,7 @@ static uint32_t aspeed_scu_calc_hpll_ast2400(AspeedSCUState 
*s)
 return hpll_ast2400_freqs[clk_25m_in][freq_select] * 100;
 }
 
-static uint32_t aspeed_scu_calc_hpll_ast2500(AspeedSCUState *s)
+static uint32_t aspeed_2500_scu_calc_hpll(AspeedSCUState *s)
 {
 uint32_t hpll_reg   = s->regs[HPLL_PARAM];
 uint32_t multiplier = 1;
@@ -357,25 +344,9 @@ static uint32_t 
aspeed_scu_calc_hpll_ast2500(AspeedSCUState *s)
 static void aspeed_scu_reset(DeviceState *dev)
 {
 AspeedSCUState *s = ASPEED_SCU(dev);
-const uint32_t *reset;
-uint32_t (*calc_hpll)(AspeedSCUState *s);
+AspeedSCUClass *asc = ASPEED_SCU_GET_CLASS(dev);
 
-switch (s->silicon_rev) {
-case AST2400_A0_SILICON_REV:
-case AST2400_A1_SILICON_REV:
-reset = ast2400_a0_resets;
-calc_hpll = aspeed_scu_calc_hpll_ast2400;
-break;
-case AST2500_A0_SILICON_REV:
-case AST2500_A1_SILICON_REV:
-reset = ast2500_a1_resets;
-calc_hpll = aspeed_scu_calc_hpll_ast2500;
-break;
-default:
-g_assert_not_reached();
-}
-
-memcpy(s->regs, reset, sizeof(s->regs));
+memcpy(s->regs, asc->resets, sizeof(s->regs));
 s->regs[SILICON_REV] = s->silicon_rev;
 s->regs[HW_STRAP1] = s->hw_strap1;
 s->regs[HW_STRAP2] = s->hw_strap2;
@@ -385,7 +356,7 @@ static void aspeed_scu_reset(DeviceState *dev)
  * All registers are set. Now compute the frequencies of the main clocks
  */
 s->clkin = aspeed_scu_get_clkin(s);
-s->hpll = calc_hpll(s);
+s->hpll = asc->calc_hpll(s);
 aspeed_scu_set_apb_freq(s);
 }
 
@@ -459,11 +430,51 @@ static co

[Qemu-devel] [PULL 08/12] aspeed/smc: Calculate checksum on normal DMA

[Peter Maydell]

From: Christian Svensson 

This patch adds the missing checksum calculation on normal DMA transfer.
According to the datasheet this is how the SMC should behave.

Verified on AST1250 that the hardware matches the behaviour.

Signed-off-by: Christian Svensson 
Reviewed-by: Joel Stanley 
Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-9-...@kaod.org
Signed-off-by: Peter Maydell 
---
 hw/ssi/aspeed_smc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/ssi/aspeed_smc.c b/hw/ssi/aspeed_smc.c
index 5c3436db5e8..9ffc7e01179 100644
--- a/hw/ssi/aspeed_smc.c
+++ b/hw/ssi/aspeed_smc.c
@@ -989,6 +989,7 @@ static void aspeed_smc_dma_rw(AspeedSMCState *s)
 s->regs[R_DMA_FLASH_ADDR] += 4;
 s->regs[R_DMA_DRAM_ADDR] += 4;
 s->regs[R_DMA_LEN] -= 4;
+s->regs[R_DMA_CHECKSUM] += data;
 }
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 01/12] hw/gpio: Add basic Aspeed GPIO model for AST2400 and AST2500

[Peter Maydell]

From: Rashmica Gupta 

GPIO pins are arranged in groups of 8 pins labeled A,B,..,Y,Z,AA,AB,AC.
(Note that the ast2400 controller only goes up to group AB).
A set has four groups (except set AC which only has one) and is
referred to by the groups it is composed of (eg ABCD,EFGH,...,YZAAAB).
Each set is accessed and controlled by a bank of 14 registers.

These registers operate on a per pin level where each bit in the register
corresponds to a pin, except for the command source registers. The command
source registers operate on a per group level where bits 24, 16, 8 and 0
correspond to each group in the set.

 eg. registers for set ABCD:
 |D7...D0|C7...C0|B7...B0|A7...A0| <- GPIOs
 |31...24|23...16|158|7.0| <- bit position

Note that there are a couple of groups that only have 4 pins.

There are two ways that this model deviates from the behaviour of the
actual controller:
(1) The only control source driving the GPIO pins in the model is the ARM
model (as there currently aren't models for the LPC or Coprocessor).

(2) None of the registers in the model are reset tolerant (needs
integration with the watchdog).

Signed-off-by: Rashmica Gupta 
Tested-by: Andrew Jeffery 
Reviewed-by: Cédric Le Goater 
Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-2-...@kaod.org
[clg: fixed missing header files
  made use of HWADDR_PRIx to fix compilation on windows ]
Signed-off-by: Cédric Le Goater 
Signed-off-by: Peter Maydell 
---
 hw/gpio/Makefile.objs |   1 +
 include/hw/gpio/aspeed_gpio.h | 100 
 hw/gpio/aspeed_gpio.c | 884 ++
 3 files changed, 985 insertions(+)
 create mode 100644 include/hw/gpio/aspeed_gpio.h
 create mode 100644 hw/gpio/aspeed_gpio.c

diff --git a/hw/gpio/Makefile.objs b/hw/gpio/Makefile.objs
index e5da0cb54fe..d305b3b24b1 100644
--- a/hw/gpio/Makefile.objs
+++ b/hw/gpio/Makefile.objs
@@ -9,3 +9,4 @@ obj-$(CONFIG_OMAP) += omap_gpio.o
 obj-$(CONFIG_IMX) += imx_gpio.o
 obj-$(CONFIG_RASPI) += bcm2835_gpio.o
 obj-$(CONFIG_NRF51_SOC) += nrf51_gpio.o
+obj-$(CONFIG_ASPEED_SOC) += aspeed_gpio.o
diff --git a/include/hw/gpio/aspeed_gpio.h b/include/hw/gpio/aspeed_gpio.h
new file mode 100644
index 000..a2deac046af
--- /dev/null
+++ b/include/hw/gpio/aspeed_gpio.h
@@ -0,0 +1,100 @@
+/*
+ *  ASPEED GPIO Controller
+ *
+ *  Copyright (C) 2017-2018 IBM Corp.
+ *
+ * This code is licensed under the GPL version 2 or later.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef ASPEED_GPIO_H
+#define ASPEED_GPIO_H
+
+#include "hw/sysbus.h"
+
+#define TYPE_ASPEED_GPIO "aspeed.gpio"
+#define ASPEED_GPIO(obj) OBJECT_CHECK(AspeedGPIOState, (obj), TYPE_ASPEED_GPIO)
+#define ASPEED_GPIO_CLASS(klass) \
+ OBJECT_CLASS_CHECK(AspeedGPIOClass, (klass), TYPE_ASPEED_GPIO)
+#define ASPEED_GPIO_GET_CLASS(obj) \
+ OBJECT_GET_CLASS(AspeedGPIOClass, (obj), TYPE_ASPEED_GPIO)
+
+#define ASPEED_GPIO_MAX_NR_SETS 8
+#define ASPEED_REGS_PER_BANK 14
+#define ASPEED_GPIO_MAX_NR_REGS (ASPEED_REGS_PER_BANK * 
ASPEED_GPIO_MAX_NR_SETS)
+#define ASPEED_GPIO_NR_PINS 228
+#define ASPEED_GROUPS_PER_SET 4
+#define ASPEED_GPIO_NR_DEBOUNCE_REGS 3
+#define ASPEED_CHARS_PER_GROUP_LABEL 4
+
+typedef struct GPIOSets GPIOSets;
+
+typedef struct GPIOSetProperties {
+uint32_t input;
+uint32_t output;
+char group_label[ASPEED_GROUPS_PER_SET][ASPEED_CHARS_PER_GROUP_LABEL];
+} GPIOSetProperties;
+
+enum GPIORegType {
+gpio_not_a_reg,
+gpio_reg_data_value,
+gpio_reg_direction,
+gpio_reg_int_enable,
+gpio_reg_int_sens_0,
+gpio_reg_int_sens_1,
+gpio_reg_int_sens_2,
+gpio_reg_int_status,
+gpio_reg_reset_tolerant,
+gpio_reg_debounce_1,
+gpio_reg_debounce_2,
+gpio_reg_cmd_source_0,
+gpio_reg_cmd_source_1,
+gpio_reg_data_read,
+gpio_reg_input_mask,
+};
+
+typedef struct AspeedGPIOReg {
+uint16_t set_idx;
+enum GPIORegType type;
+ } AspeedGPIOReg;
+
+typedef struct  AspeedGPIOClass {
+SysBusDevice parent_obj;
+const GPIOSetProperties *props;
+uint32_t nr_gpio_pins;
+uint32_t nr_gpio_sets;
+uint32_t gap;
+const AspeedGPIOReg *reg_table;
+}  AspeedGPIOClass;
+
+typedef struct AspeedGPIOState {
+/*  */
+SysBusDevice parent;
+
+/*< public >*/
+MemoryRegion iomem;
+int pending;
+qemu_irq irq;
+qemu_irq gpios[ASPEED_GPIO_NR_PINS];
+
+/* Parallel GPIO Registers */
+uint32_t debounce_regs[ASPEED_GPIO_NR_DEBOUNCE_REGS];
+struct GPIOSets {
+uint32_t data_value; /* Reflects pin values */
+uint32_t data_read; /* Contains last value written to data value */
+uint32_t direction;
+uint32_t int_enable;
+uint32_t int_sens_0;
+uint32_t int_sens_1;
+uint32_t int_sens_2;
+uint32_t int_status;
+uint32_t reset_tol;
+uint32_t cmd_source_0;
+uint32_t cmd_source_1;
+uint32_t debounce_1;
+uint32_t debounce_2;
+uint32_t input_mask;
+} sets[

[Qemu-devel] [PULL 00/12] target-arm queue

[Peter Maydell]

target-arm queue: mostly aspeed changes from Cédric.

thanks
-- PMM

The following changes since commit 85182c96de61f0b600bbe834d5a23e713162e892:

  Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20190912a' 
into staging (2019-09-13 14:37:48 +0100)

are available in the Git repository at:

  https://git.linaro.org/people/pmaydell/qemu-arm.git 
tags/pull-target-arm-20190913

for you to fetch changes up to 27a296fce9821e3608d537756cffa6e43a46df3b:

  qemu-ga: Convert invocation documentation to rST (2019-09-13 16:05:01 +0100)


target-arm queue:
 * aspeed: add a GPIO controller to the SoC
 * aspeed: Various refactorings
 * aspeed: Improve DMA controller modelling
 * atomic_template: fix indentation in GEN_ATOMIC_HELPER
 * qemu-ga: Convert invocation documentation to rST


Christian Svensson (1):
  aspeed/smc: Calculate checksum on normal DMA

Cédric Le Goater (7):
  aspeed: Remove unused SoC definitions
  aspeed: Use consistent typenames
  aspeed/smc: Add support for DMAs
  aspeed/smc: Add DMA calibration settings
  aspeed/smc: Inject errors in DMA checksum
  aspeed/scu: Introduce per-SoC SCU types
  aspeed/scu: Introduce a aspeed_scu_get_apb_freq() routine

Emilio G. Cota (1):
  atomic_template: fix indentation in GEN_ATOMIC_HELPER

Peter Maydell (1):
  qemu-ga: Convert invocation documentation to rST

Rashmica Gupta (2):
  hw/gpio: Add basic Aspeed GPIO model for AST2400 and AST2500
  aspeed: add a GPIO controller to the SoC

 Makefile  |  24 +-
 hw/gpio/Makefile.objs |   1 +
 accel/tcg/atomic_template.h   |   2 +-
 include/hw/arm/aspeed_soc.h   |   4 +-
 include/hw/gpio/aspeed_gpio.h | 100 +
 include/hw/misc/aspeed_scu.h  |  21 +-
 include/hw/ssi/aspeed_smc.h   |   7 +
 hw/arm/aspeed.c   |   2 +
 hw/arm/aspeed_soc.c   |  63 ++-
 hw/gpio/aspeed_gpio.c | 884 ++
 hw/misc/aspeed_scu.c  | 102 ++---
 hw/ssi/aspeed_smc.c   | 335 +++-
 hw/timer/aspeed_timer.c   |   3 +-
 MAINTAINERS   |   2 +-
 docs/conf.py  |  18 +-
 docs/interop/conf.py  |   7 +
 docs/interop/index.rst|   1 +
 docs/interop/qemu-ga.rst  | 133 +++
 qemu-doc.texi |   5 -
 qemu-ga.texi  | 137 ---
 20 files changed, 1585 insertions(+), 266 deletions(-)
 create mode 100644 include/hw/gpio/aspeed_gpio.h
 create mode 100644 hw/gpio/aspeed_gpio.c
 create mode 100644 docs/interop/qemu-ga.rst
 delete mode 100644 qemu-ga.texi

[Qemu-devel] [PULL 03/12] aspeed: Remove unused SoC definitions

[Peter Maydell]

From: Cédric Le Goater 

There are no QEMU Aspeed machines using the SoCs "ast2400-a0" or
"ast2400".

Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-4-...@kaod.org
Reviewed-by: Peter Maydell 
Signed-off-by: Peter Maydell 
---
 hw/arm/aspeed_soc.c | 26 --
 1 file changed, 26 deletions(-)

diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
index 04480875d0d..3aa73d2438e 100644
--- a/hw/arm/aspeed_soc.c
+++ b/hw/arm/aspeed_soc.c
@@ -118,19 +118,6 @@ static const char *aspeed_soc_ast2500_typenames[] = {
 
 static const AspeedSoCInfo aspeed_socs[] = {
 {
-.name = "ast2400-a0",
-.cpu_type = ARM_CPU_TYPE_NAME("arm926"),
-.silicon_rev  = AST2400_A0_SILICON_REV,
-.sram_size= 0x8000,
-.spis_num = 1,
-.fmc_typename = "aspeed.smc.fmc",
-.spi_typename = aspeed_soc_ast2400_typenames,
-.gpio_typename = "aspeed.gpio-ast2400",
-.wdts_num = 2,
-.irqmap   = aspeed_soc_ast2400_irqmap,
-.memmap   = aspeed_soc_ast2400_memmap,
-.num_cpus = 1,
-}, {
 .name = "ast2400-a1",
 .cpu_type = ARM_CPU_TYPE_NAME("arm926"),
 .silicon_rev  = AST2400_A1_SILICON_REV,
@@ -143,19 +130,6 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .irqmap   = aspeed_soc_ast2400_irqmap,
 .memmap   = aspeed_soc_ast2400_memmap,
 .num_cpus = 1,
-}, {
-.name = "ast2400",
-.cpu_type = ARM_CPU_TYPE_NAME("arm926"),
-.silicon_rev  = AST2400_A0_SILICON_REV,
-.sram_size= 0x8000,
-.spis_num = 1,
-.fmc_typename = "aspeed.smc.fmc",
-.spi_typename = aspeed_soc_ast2400_typenames,
-.gpio_typename = "aspeed.gpio-ast2400",
-.wdts_num = 2,
-.irqmap   = aspeed_soc_ast2400_irqmap,
-.memmap   = aspeed_soc_ast2400_memmap,
-.num_cpus = 1,
 }, {
 .name = "ast2500-a1",
 .cpu_type = ARM_CPU_TYPE_NAME("arm1176"),
-- 
2.20.1

[Qemu-devel] [PULL 04/12] aspeed: Use consistent typenames

[Peter Maydell]

From: Cédric Le Goater 

Improve the naming of the different controller models to ease their
generation when initializing the SoC. The rename of the SMC types is
breaking migration compatibility.

Signed-off-by: Cédric Le Goater 
Message-id: 20190904070506.1052-5-...@kaod.org
Reviewed-by: Peter Maydell 
Signed-off-by: Peter Maydell 
---
 include/hw/arm/aspeed_soc.h |  3 ---
 hw/arm/aspeed_soc.c | 25 -
 hw/ssi/aspeed_smc.c | 12 ++--
 3 files changed, 18 insertions(+), 22 deletions(-)

diff --git a/include/hw/arm/aspeed_soc.h b/include/hw/arm/aspeed_soc.h
index a56effebc16..ab5052b12cb 100644
--- a/include/hw/arm/aspeed_soc.h
+++ b/include/hw/arm/aspeed_soc.h
@@ -61,9 +61,6 @@ typedef struct AspeedSoCInfo {
 uint32_t silicon_rev;
 uint64_t sram_size;
 int spis_num;
-const char *fmc_typename;
-const char **spi_typename;
-const char *gpio_typename;
 int wdts_num;
 const int *irqmap;
 const hwaddr *memmap;
diff --git a/hw/arm/aspeed_soc.c b/hw/arm/aspeed_soc.c
index 3aa73d2438e..25dbc409d35 100644
--- a/hw/arm/aspeed_soc.c
+++ b/hw/arm/aspeed_soc.c
@@ -112,10 +112,6 @@ static const int aspeed_soc_ast2400_irqmap[] = {
 
 #define aspeed_soc_ast2500_irqmap aspeed_soc_ast2400_irqmap
 
-static const char *aspeed_soc_ast2400_typenames[] = { "aspeed.smc.spi" };
-static const char *aspeed_soc_ast2500_typenames[] = {
-"aspeed.smc.ast2500-spi1", "aspeed.smc.ast2500-spi2" };
-
 static const AspeedSoCInfo aspeed_socs[] = {
 {
 .name = "ast2400-a1",
@@ -123,9 +119,6 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .silicon_rev  = AST2400_A1_SILICON_REV,
 .sram_size= 0x8000,
 .spis_num = 1,
-.fmc_typename = "aspeed.smc.fmc",
-.spi_typename = aspeed_soc_ast2400_typenames,
-.gpio_typename = "aspeed.gpio-ast2400",
 .wdts_num = 2,
 .irqmap   = aspeed_soc_ast2400_irqmap,
 .memmap   = aspeed_soc_ast2400_memmap,
@@ -136,9 +129,6 @@ static const AspeedSoCInfo aspeed_socs[] = {
 .silicon_rev  = AST2500_A1_SILICON_REV,
 .sram_size= 0x9000,
 .spis_num = 2,
-.fmc_typename = "aspeed.smc.ast2500-fmc",
-.spi_typename = aspeed_soc_ast2500_typenames,
-.gpio_typename = "aspeed.gpio-ast2500",
 .wdts_num = 3,
 .irqmap   = aspeed_soc_ast2500_irqmap,
 .memmap   = aspeed_soc_ast2500_memmap,
@@ -158,6 +148,12 @@ static void aspeed_soc_init(Object *obj)
 AspeedSoCState *s = ASPEED_SOC(obj);
 AspeedSoCClass *sc = ASPEED_SOC_GET_CLASS(s);
 int i;
+char socname[8];
+char typename[64];
+
+if (sscanf(sc->info->name, "%7s", socname) != 1) {
+g_assert_not_reached();
+}
 
 for (i = 0; i < sc->info->num_cpus; i++) {
 object_initialize_child(obj, "cpu[*]", OBJECT(&s->cpu[i]),
@@ -190,14 +186,16 @@ static void aspeed_soc_init(Object *obj)
 sysbus_init_child_obj(obj, "i2c", OBJECT(&s->i2c), sizeof(s->i2c),
   TYPE_ASPEED_I2C);
 
+snprintf(typename, sizeof(typename), "aspeed.fmc-%s", socname);
 sysbus_init_child_obj(obj, "fmc", OBJECT(&s->fmc), sizeof(s->fmc),
-  sc->info->fmc_typename);
+  typename);
 object_property_add_alias(obj, "num-cs", OBJECT(&s->fmc), "num-cs",
   &error_abort);
 
 for (i = 0; i < sc->info->spis_num; i++) {
+snprintf(typename, sizeof(typename), "aspeed.spi%d-%s", i + 1, 
socname);
 sysbus_init_child_obj(obj, "spi[*]", OBJECT(&s->spi[i]),
-  sizeof(s->spi[i]), sc->info->spi_typename[i]);
+  sizeof(s->spi[i]), typename);
 }
 
 sysbus_init_child_obj(obj, "sdmc", OBJECT(&s->sdmc), sizeof(s->sdmc),
@@ -226,8 +224,9 @@ static void aspeed_soc_init(Object *obj)
 sysbus_init_child_obj(obj, "xdma", OBJECT(&s->xdma), sizeof(s->xdma),
   TYPE_ASPEED_XDMA);
 
+snprintf(typename, sizeof(typename), "aspeed.gpio-%s", socname);
 sysbus_init_child_obj(obj, "gpio", OBJECT(&s->gpio), sizeof(s->gpio),
-  sc->info->gpio_typename);
+  typename);
 }
 
 static void aspeed_soc_realize(DeviceState *dev, Error **errp)
diff --git a/hw/ssi/aspeed_smc.c b/hw/ssi/aspeed_smc.c
index 9f3cff5fb6f..f4f7c181830 100644
--- a/hw/ssi/aspeed_smc.c
+++ b/hw/ssi/aspeed_smc.c
@@ -190,7 +190,7 @@ static const AspeedSegments aspeed_segments_ast2500_spi2[] 
= {
 
 static const AspeedSMCController controllers[] = {
 {
-.name  = "aspeed.smc.smc",
+.name  = "aspeed.smc-ast2400",
 .r_conf= R_CONF,
 .r_ce_ctrl = R_CE_CTRL,
 .r_ctrl0   = R_CTRL0,
@@ -203,7 +203,7 @@ static const AspeedSMCController controllers[] = {
 .has_dma   = false,
 .nregs = A

Re: [Qemu-devel] [PATCH v8 01/13] vfio: KABI for migration interface

[Alex Williamson]

On Thu, 12 Sep 2019 23:00:03 +
"Tian, Kevin"  wrote:

> > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > Sent: Thursday, September 12, 2019 10:41 PM
> > 
> > On Tue, 3 Sep 2019 06:57:27 +
> > "Tian, Kevin"  wrote:
> >   
> > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > Sent: Saturday, August 31, 2019 12:33 AM
> > > >
> > > > On Fri, 30 Aug 2019 08:06:32 +
> > > > "Tian, Kevin"  wrote:
> > > >  
> > > > > > From: Tian, Kevin
> > > > > > Sent: Friday, August 30, 2019 3:26 PM
> > > > > >  
> > > > > [...]  
> > > > > > > How does QEMU handle the fact that IOVAs are potentially  
> > dynamic  
> > > > while  
> > > > > > > performing the live portion of a migration?  For example, each  
> > time a  
> > > > > > > guest driver calls dma_map_page() or dma_unmap_page(), a
> > > > > > > MemoryRegionSection pops in or out of the AddressSpace for the  
> > device  
> > > > > > > (I'm assuming a vIOMMU where the device AddressSpace is not
> > > > > > > system_memory).  I don't see any QEMU code that intercepts that  
> > > > change  
> > > > > > > in the AddressSpace such that the IOVA dirty pfns could be  
> > recorded and  
> > > > > > > translated to GFNs.  The vendor driver can't track these beyond  
> > getting  
> > > > > > > an unmap notification since it only knows the IOVA pfns, which  
> > can be  
> > > > > > > re-used with different GFN backing.  Once the DMA mapping is  
> > torn  
> > > > down,  
> > > > > > > it seems those dirty pfns are lost in the ether.  If this works 
> > > > > > > in  
> > QEMU,  
> > > > > > > please help me find the code that handles it.  
> > > > > >
> > > > > > I'm curious about this part too. Interestingly, I didn't find any  
> > log_sync  
> > > > > > callback registered by emulated devices in Qemu. Looks dirty pages
> > > > > > by emulated DMAs are recorded in some implicit way. But KVM  
> > always  
> > > > > > reports dirty page in GFN instead of IOVA, regardless of the  
> > presence of  
> > > > > > vIOMMU. If Qemu also tracks dirty pages in GFN for emulated DMAs
> > > > > >  (translation can be done when DMA happens), then we don't need
> > > > > > worry about transient mapping from IOVA to GFN. Along this way  
> > we  
> > > > > > also want GFN-based dirty bitmap being reported through VFIO,
> > > > > > similar to what KVM does. For vendor drivers, it needs to translate
> > > > > > from IOVA to HVA to GFN when tracking DMA activities on VFIO
> > > > > > devices. IOVA->HVA is provided by VFIO. for HVA->GFN, it can be
> > > > > > provided by KVM but I'm not sure whether it's exposed now.
> > > > > >  
> > > > >
> > > > > HVA->GFN can be done through hva_to_gfn_memslot in kvm_host.h.  
> > > >
> > > > I thought it was bad enough that we have vendor drivers that depend  
> > on  
> > > > KVM, but designing a vfio interface that only supports a KVM interface
> > > > is more undesirable.  I also note without comment that  
> > gfn_to_memslot()  
> > > > is a GPL symbol.  Thanks,  
> > >
> > > yes it is bad, but sometimes inevitable. If you recall our discussions
> > > back to 3yrs (when discussing the 1st mdev framework), there were  
> > similar  
> > > hypervisor dependencies in GVT-g, e.g. querying gpa->hpa when
> > > creating some shadow structures. gpa->hpa is definitely hypervisor
> > > specific knowledge, which is easy in KVM (gpa->hva->hpa), but needs
> > > hypercall in Xen. but VFIO already makes assumption based on KVM-
> > > only flavor when implementing vfio_{un}pin_page_external.  
> > 
> > Where's the KVM assumption there?  The MAP_DMA ioctl takes an IOVA
> > and
> > HVA.  When an mdev vendor driver calls vfio_pin_pages(), we GUP the HVA
> > to get an HPA and provide an array of HPA pfns back to the caller.  The
> > other vGPU mdev vendor manages to make use of this without KVM... the
> > KVM interface used by GVT-g is GPL-only.  
> 
> To be clear it's the assumption on the host-based hypervisors e.g. KVM.
> GUP is a perfect example, which doesn't work for Xen since DomU's
> memory doesn't belong to Dom0. VFIO in Dom0 has to find the HPA
> through Xen specific hypercalls.

VFIO does not assume a hypervisor at all.  Yes, it happens to work well
with a host-based hypervisor like KVM were we can simply use GUP, but
I'd hardly call using the standard mechanism to pin a user page and get
the pfn within the Linux kernel a KVM assumption.  The fact that Dom0
Xen requires work here while KVM does not does is not an equivalency to
VFIO assuming KVM.  Thanks,

Alex
 
> > > So GVT-g
> > > has to maintain an internal abstraction layer to support both Xen and
> > > KVM. Maybe someday we will re-consider introducing some hypervisor
> > > abstraction layer in VFIO, if this issue starts to hurt other devices and
> > > Xen guys are willing to support VFIO.  
> > 
> > Once upon a time, we had a KVM specific device assignment interface,
> > ie. legacy KVM devie assignment.  We developed VFIO specifically to get
> > KVM out 

Re: [Qemu-devel] [PATCH v5 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[Vladimir Sementsov-Ogievskiy]

13.09.2019 18:28, Maxim Levitsky wrote:
> This fixes subtle corruption introduced by luks threaded encryption
> in commit 8ac0f15f335
> 
> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922
> 
> The corruption happens when we do a write that
> * writes to two or more unallocated clusters at once
> * doesn't fully cover the first sector
> * doesn't fully cover the last sector
> 
> In this case, when allocating the new clusters we COW both areas
> prior to the write and after the write, and we encrypt them.
> 
> The above mentioned commit accidentally made it so we encrypt the
> second COW area using the physical cluster offset of the first area.
> 
> The problem is that offset_in_cluster in do_perform_cow_encrypt
> can be larger that the cluster size, thus cluster_offset
> will no longer point to the start of the cluster at which encrypted
> area starts.
> 
> Next patch in this series will refactor the code to avoid all these
> assumptions.
> 
> In the bugreport that was triggered by rebasing a luks image to new,
> zero filled base, which lot of such writes, and causes some files
> with zero areas to contain garbage there instead.
> But as described above it can happen elsewhere as well
> 
> 
> Signed-off-by: Maxim Levitsky 

Reviewed-by: Vladimir Sementsov-Ogievskiy 

> ---
>   block/qcow2-cluster.c | 7 ---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> index dcacd3c450..bfeb0241d7 100644
> --- a/block/qcow2-cluster.c
> +++ b/block/qcow2-cluster.c
> @@ -474,9 +474,10 @@ static bool coroutine_fn 
> do_perform_cow_encrypt(BlockDriverState *bs,
>   assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
>   assert((bytes & ~BDRV_SECTOR_MASK) == 0);
>   assert(s->crypto);
> -if (qcow2_co_encrypt(bs, cluster_offset,
> - src_cluster_offset + offset_in_cluster,
> - buffer, bytes) < 0) {
> +if (qcow2_co_encrypt(bs,
> +start_of_cluster(s, cluster_offset + offset_in_cluster),
> +src_cluster_offset + offset_in_cluster,
> +buffer, bytes) < 0) {
>   return false;
>   }
>   }
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v11 00/14] backup-top filter driver for backup

[Vladimir Sementsov-Ogievskiy]

10.09.2019 13:23, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> These series introduce backup-top driver. It's a filter-node, which
> do copy-before-write operation. Mirror uses filter-node for handling
> guest writes, let's move to filter-node (from write-notifiers) for
> backup too.
> 
> v11: based on Kevin's block branch

Now based on master

> 
> 02,03: Add Max's r-b
> 04: - improve comments
>  - rebase on dropped backup_drain
>  - s/job/bcs/ in trace events
> 05: one hunk dropped due to rebasing
>  on dropped backup_drain, still,
>  keep Max's r-b
> 06: rebased on 04 changes, keep Max's r-b
> 08,13,14: add Max's r-b
> 
> Based-on: git://repo.or.cz/qemu/kevin.git block
> 
> Vladimir Sementsov-Ogievskiy (14):
>block/backup: fix backup_cow_with_offload for last cluster
>block/backup: split shareable copying part from backup_do_cow
>block/backup: improve comment about image fleecing
>block/backup: introduce BlockCopyState
>block/backup: fix block-comment style
>block: move block_copy from block/backup.c to separate file
>block: teach bdrv_debug_breakpoint skip filters with backing
>iotests: prepare 124 and 257 bitmap querying for backup-top filter
>iotests: 257: drop unused Drive.device field
>iotests: 257: drop device_add
>block/io: refactor wait_serialising_requests
>block: add lock/unlock range functions
>block: introduce backup-top filter driver
>block/backup: use backup-top instead of write notifiers
> 
>   qapi/block-core.json  |   8 +-
>   block/backup-top.h|  37 ++
>   include/block/block-copy.h|  84 
>   include/block/block_int.h |   5 +
>   block.c   |  34 +-
>   block/backup-top.c| 240 
>   block/backup.c| 440 -
>   block/block-copy.c| 337 
>   block/io.c|  68 +++-
>   block/replication.c   |   2 +-
>   blockdev.c|   1 +
>   block/Makefile.objs   |   3 +
>   block/trace-events|  14 +-
>   tests/qemu-iotests/056|   8 +-
>   tests/qemu-iotests/124|  83 ++--
>   tests/qemu-iotests/257|  91 ++---
>   tests/qemu-iotests/257.out| 714 ++
>   tests/qemu-iotests/iotests.py |  27 ++
>   18 files changed, 1278 insertions(+), 918 deletions(-)
>   create mode 100644 block/backup-top.h
>   create mode 100644 include/block/block-copy.h
>   create mode 100644 block/backup-top.c
>   create mode 100644 block/block-copy.c
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v2 15/16] docs/devel/qapi-code-gen: Improve QAPI schema language doc

[Markus Armbruster]

Eric Blake  writes:

> On 9/10/19 1:37 AM, Markus Armbruster wrote:
>> We document the language by giving patterns of valid JSON objects.
>> The patterns contain placeholders we don't define anywhere; their
>> names have to speak for themselves.  I guess they do, but I'd prefer a
>> bit more rigor.  Provide a grammar instead, and rework the text
>> accordingly.
>> 
>> Documentation for QAPI schema conditionals (commit 967c885108,
>> 6cc32b0e14, 87adbbffd4..3e270dcacc) and feature flags (commit
>> 6a8c0b5102) was bolted on.  The sections documenting types, commands
>> and events don't mention them.  Section "Features" and "Configuring
>> the schema" then provide additional syntax for types, commands and
>> events.  I hate that.  Fix the sections documenting types, commands
>> and events to provide accurate syntax, and point to "Features" and
>> "Configuring the schema" for details.
>> 
>> We talk about "(top-level) expressions other than include and pragma".
>> Adopt more convenient terminology: a (top-level) expression is either
>> a directive (include or pragma) or a definition (anything else).
>> 
>> Avoid the terms "dictionary" and "key".  Stick to JSON terminology
>> "object" and "member name" instead.
>> 
>> While there, make spacing more consistent.
>> 
>> Signed-off-by: Markus Armbruster 
>> ---
>>  docs/devel/qapi-code-gen.txt | 592 ++-
>>  1 file changed, 369 insertions(+), 223 deletions(-)
>
>>  
>> -The order of keys within JSON objects does not matter unless
>> +* Production rules look like non-terminal = expression
>> +* Concatenation: Expression A B matches expression A, then B
>> +* Alternation: Expression A | B matches expression A or B
>> +* Repetition: Expression A... matches zero or more occurences of
>
> occurrences

Will fix.

>> +  expression A; expression A, ... likewise, but separated by ,
>
> worth calling out that trailing , are not allowed?

Doesn't "separated by" imply that?

> Is the 'expression A;' a copy-paste from RFC text, irrelevant to our
> usage here?

What about

* Repetition: Expression A... matches zero or more occurences of
  expression A
* Repetition: Expression A, ... matches zero or more occurences of
  expression A separated by ,

>> +* Grouping: Expression ( A ) matches expression A
>> +* JSON's structural characters are terminals: { } [ ] : ,
>> +* JSON's literal names are terminals: false true null
>> +* String literals enclosed in 'single quotes' are terminal, and match
>> +  this JSON string, with a leading '*' stripped off
>> +* When JSON object member's name starts with '*', the member is
>> +  optional.
>> +* The symbol STRING is a terminal, and matches any JSON string
>> +* The symbol BOOL is a terminal, and matches JSON false or true
>> +* ALL-CAPS words other than STRING are non-terminals
>> +
>> +The order of members within JSON objects does not matter unless
>>  explicitly noted.
>>  
>
>> +A QAPI schema consists of a series of top-level expressions:
>> +
>> +SCHEMA = TOP-LEVEL-EXPR...
>> +
>> +The top-level expressions are all JSON objects.  Their order does not
>> +matter.
>
> Is that always true for all directives?

Yes from a purely semantic point of view.

No when you look at the generated text: that's in source order.  Should
not matter for C.  Does matter for documentation.

See also discussion of previous patch.

> Would it be worth reformulating as something like:
>
> SCHEMA = DIRECTIVE... DEFINITION...
>
> allowing zero-or-more of either, but where directives come first?

Language change.  Not sure anything outside tests would break.  But why
bother?

>> +
>> +A top-level expressions is either a directive or a definition:
>> +
>> +TOP-LEVEL-EXPR = DIRECTIVE | DEFINITION
>> +
>> +There are two kinds of directives and six kinds of definitions:
>> +
>> +DIRECTIVE = INCLUDE | PRAGMA
>> +DEFINITION = ENUM | STRUCT | UNION | ALTERNATE | COMMAND | EVENT
>> +
>> +These are discussed in detail below.
>>  
>>  
>
>>  === Pragma directives ===
>>  
>> -Usage: { 'pragma': DICT }
>> +Syntax:
>> +PRAGMA = { 'pragma': { '*doc-required': BOOL },
>> +   '*returns-whitelist': [ STRING, ... ],
>> +   '*name-case-whitelist': [ STRING, ... ] }
>
> This has matched {}, but looks wrong.  I think you meant
>
> PRAGMA = { 'pragma': { '*doc-required': BOOL,
>'*returns-whitelist': [ STRING, ... ],
>'*name-case-whitelist': [ STRING, ... ] } }

Good catch.

>>  === Enumeration types ===
>>  
>> -Usage: { 'enum': STRING, 'data': ARRAY-OF-STRING }
>> -   { 'enum': STRING, '*prefix': STRING, 'data': ARRAY-OF-STRING }
>> +Syntax:
>> +ENUM = { 'enum': STRING,
>> + 'data': [ ENUM-VALUE, ... ],
>> + '*prefix': STRING,
>> + '*if': COND }
>> +ENUM-VALUE = STRING
>> +   | { 'name': STRING, '*if': COND }
>>  
>
>> +=== Type references and array types ===
>> +

Re: [Qemu-devel] [PATCH v3 3/3] xen: perform XenDevice clean-up in XenBus watch handler

[Anthony PERARD]

On Fri, Sep 13, 2019 at 09:21:58AM +0100, Paul Durrant wrote:
> Cleaning up offline XenDevice objects directly in
> xen_device_backend_changed() is dangerous as xen_device_unrealize() will
> modify the watch list that is being walked. Even the QLIST_FOREACH_SAFE()
> used in notifier_list_notify() is insufficient as *two* notifiers (for
> the frontend and backend watches) are removed, thus potentially rendering
> the 'next' pointer unsafe.
> 
> The solution is to use the XenBus backend_watch handler to do the clean-up
> instead, as it is invoked whilst walking a separate watch list.
> 
> This patch therefore adds a new 'inactive_devices' list to XenBus, to which
> offline devices are added by xen_device_backend_changed(). The XenBus
> backend_watch registration is also changed to not only invoke
> xen_bus_enumerate() but also a new xen_bus_cleanup() function, which will
> walk 'inactive_devices' and perform the necessary actions.
> For safety an extra 'online' check is also added to xen_bus_type_enumerate()
> to make sure that no attempt is made to create a new XenDevice object for a
> backend that is offline.
> 
> NOTE: This patch also includes some cosmetic changes:
>   - substitute the local variable name 'backend_state'
> in xen_bus_type_enumerate() with 'state', since there
> is no ambiguity with any other state in that context.
>   - change xen_device_state_is_active() to
> xen_device_frontend_is_active() (and pass a XenDevice directly)
> since the state tests contained therein only apply to a frontend.
>   - use 'state' rather then 'xendev->backend_state' in
> xen_device_backend_changed() to shorten the code.
> 
> Signed-off-by: Paul Durrant 
> ---
> Cc: Stefano Stabellini 
> Cc: Anthony Perard 
> 
> v3:
>  - s/offline_devices/inactive_devices/g
>  - Also add an 'inactive' boolean to XenDevice which is set when the
>device is added to the inactive list so we really can make sure that it
>doesn't happen more than once

Reviewed-by: Anthony PERARD 

Thanks,

-- 
Anthony PERARD

[Qemu-devel] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

[Maxim Levitsky]

Change do_perform_cow_encrypt and its callee qcow2_co_encrypt
to just receive full host and guest offsets and in pariticular
remove the offset_in_cluster parameter of do_perform_cow_encrypt,
since it is misleading, because that offset can be larger than
cluster size currently.

Also document the qcow2_co_encrypt arguments to prevent
that bug from happening again

Signed-off-by: Maxim Levitsky 
---
 block/qcow2-cluster.c | 30 -
 block/qcow2-threads.c | 62 ++-
 block/qcow2.c |  5 ++--
 block/qcow2.h |  8 +++---
 4 files changed, 73 insertions(+), 32 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index bfeb0241d7..e87a4637fd 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -463,21 +463,21 @@ static int coroutine_fn 
do_perform_cow_read(BlockDriverState *bs,
 }
 
 static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
-uint64_t src_cluster_offset,
-uint64_t cluster_offset,
-unsigned offset_in_cluster,
+uint64_t guest_offset,
+uint64_t host_offset,
 uint8_t *buffer,
 unsigned bytes)
 {
 if (bytes && bs->encrypted) {
 BDRVQcow2State *s = bs->opaque;
-assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
-assert((bytes & ~BDRV_SECTOR_MASK) == 0);
+
+assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
+assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
+assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 assert(s->crypto);
-if (qcow2_co_encrypt(bs,
-start_of_cluster(s, cluster_offset + offset_in_cluster),
-src_cluster_offset + offset_in_cluster,
-buffer, bytes) < 0) {
+
+if (qcow2_co_encrypt(bs, host_offset, guest_offset,
+buffer, bytes) < 0) {
 return false;
 }
 }
@@ -891,11 +891,15 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta 
*m)
 
 /* Encrypt the data if necessary before writing it */
 if (bs->encrypted) {
-if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-start->offset, start_buffer,
+if (!do_perform_cow_encrypt(bs,
+m->offset + start->offset,
+m->alloc_offset + start->offset,
+start_buffer,
 start->nb_bytes) ||
-!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-end->offset, end_buffer, end->nb_bytes)) {
+!do_perform_cow_encrypt(bs,
+m->offset + end->offset,
+m->alloc_offset + end->offset,
+end_buffer, end->nb_bytes)) {
 ret = -EIO;
 goto fail;
 }
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 3b1e63fe41..9646243a9b 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -234,15 +234,15 @@ static int qcow2_encdec_pool_func(void *opaque)
 }
 
 static int coroutine_fn
-qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
-  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc func)
+qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
+uint64_t guest_offset, void *buf, size_t len,
+Qcow2EncDecFunc func)
 {
 BDRVQcow2State *s = bs->opaque;
+
 Qcow2EncDecData arg = {
 .block = s->crypto,
-.offset = s->crypt_physical_offset ?
-  file_cluster_offset + offset_into_cluster(s, offset) :
-  offset,
+.offset = s->crypt_physical_offset ? host_offset : guest_offset,
 .buf = buf,
 .len = len,
 .func = func,
@@ -251,18 +251,54 @@ qcow2_co_encdec(BlockDriverState *bs, uint64_t 
file_cluster_offset,
 return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
 }
 
+
+/*
+ * qcow2_co_encrypt()
+ *
+ * Encrypts one or more contiguous aligned sectors
+ *
+ * @host_offset - underlying storage offset of the first sector of the
+ * data to be encrypted
+
+ * @guest_offset - guest (virtual) offset of the first sector of the
+ * data to be encrypted
+ *
+ * @buf - buffer with the data to encrypt, that after encryption
+ *will be written to the underlying storage device at
+ *@host_offset
+ *
+ * @len - length of the buffer (must be a BDRV_SECTOR_SIZE multiple)
+ *
+ * Depending on the encryption method, @host_cluster_offset and/or 
@guest_offset
+ * may be used for ge

[Qemu-devel] [PATCH v5 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[Maxim Levitsky]

This fixes subtle corruption introduced by luks threaded encryption
in commit 8ac0f15f335

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922

The corruption happens when we do a write that
   * writes to two or more unallocated clusters at once
   * doesn't fully cover the first sector
   * doesn't fully cover the last sector

In this case, when allocating the new clusters we COW both areas
prior to the write and after the write, and we encrypt them.

The above mentioned commit accidentally made it so we encrypt the
second COW area using the physical cluster offset of the first area.

The problem is that offset_in_cluster in do_perform_cow_encrypt
can be larger that the cluster size, thus cluster_offset
will no longer point to the start of the cluster at which encrypted
area starts.

Next patch in this series will refactor the code to avoid all these
assumptions.

In the bugreport that was triggered by rebasing a luks image to new,
zero filled base, which lot of such writes, and causes some files
with zero areas to contain garbage there instead.
But as described above it can happen elsewhere as well


Signed-off-by: Maxim Levitsky 
---
 block/qcow2-cluster.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index dcacd3c450..bfeb0241d7 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -474,9 +474,10 @@ static bool coroutine_fn 
do_perform_cow_encrypt(BlockDriverState *bs,
 assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
 assert((bytes & ~BDRV_SECTOR_MASK) == 0);
 assert(s->crypto);
-if (qcow2_co_encrypt(bs, cluster_offset,
- src_cluster_offset + offset_in_cluster,
- buffer, bytes) < 0) {
+if (qcow2_co_encrypt(bs,
+start_of_cluster(s, cluster_offset + offset_in_cluster),
+src_cluster_offset + offset_in_cluster,
+buffer, bytes) < 0) {
 return false;
 }
 }
-- 
2.17.2

[Qemu-devel] [PATCH v5 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

[Maxim Levitsky]

Commit 8ac0f15f335 accidently broke the COW of non changed areas
of newly allocated clusters, when the write spans multiple clusters,
and needs COW both prior and after the write.
This results in 'after' COW area being encrypted with wrong
sector address, which render it corrupted.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922

CC: qemu-stable 

V2: grammar, spelling and code style fixes.
V3: more fixes after the review.
V4: addressed review comments from Max Reitz,
and futher refactored the qcow2_co_encrypt to just take full host and guest 
offset
which simplifies everything.

V5: reworked the patches so one of them fixes the bug
only and other one is just refactoring

Best regards,
Maxim Levitsky

Maxim Levitsky (3):
  Fix qcow2+luks corruption introduced by commit 8ac0f15f335
  block/qcow2: refactor threaded encryption code
  qemu-iotests: Add test for bz #1745922

 block/qcow2-cluster.c  | 29 +++-
 block/qcow2-threads.c  | 62 --
 block/qcow2.c  |  5 ++-
 block/qcow2.h  |  8 ++--
 tests/qemu-iotests/263 | 91 ++
 tests/qemu-iotests/263.out | 40 +
 tests/qemu-iotests/group   |  2 +
 7 files changed, 206 insertions(+), 31 deletions(-)
 create mode 100755 tests/qemu-iotests/263
 create mode 100644 tests/qemu-iotests/263.out

-- 
2.17.2

[Qemu-devel] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

[Maxim Levitsky]

Signed-off-by: Maxim Levitsky 
---
 tests/qemu-iotests/263 | 91 ++
 tests/qemu-iotests/263.out | 40 +
 tests/qemu-iotests/group   |  2 +
 3 files changed, 133 insertions(+)
 create mode 100755 tests/qemu-iotests/263
 create mode 100644 tests/qemu-iotests/263.out

diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
new file mode 100755
index 00..d2c030fae9
--- /dev/null
+++ b/tests/qemu-iotests/263
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# Test encrypted write that crosses cluster boundary of two unallocated 
clusters
+# Based on 188
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=mlevi...@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=1M
+
+SECRET="secret,id=sec0,data=astrochicken"
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+
+_run_test()
+{
+   echo "== reading the whole image =="
+   $QEMU_IO --object $SECRET -c "read -P 0 0 $size" --image-opts "$1" | 
_filter_qemu_io | _filter_testdir
+
+   echo
+   echo "== write two 512 byte sectors on a cluster boundary =="
+   $QEMU_IO --object $SECRET -c "write -P 0xAA 0xFE00 0x400" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+
+   echo
+   echo "== verify that the rest of the image is not changed =="
+   $QEMU_IO --object $SECRET -c "read -P 0x00 0x0 0xFE00" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+   $QEMU_IO --object $SECRET -c "read -P 0xAA 0x0FE00 0x400" --image-opts 
"$1" | _filter_qemu_io | _filter_testdir
+   $QEMU_IO --object $SECRET -c "read -P 0x00 0x10200 0xEFE00" 
--image-opts "$1" | _filter_qemu_io | _filter_testdir
+
+}
+
+
+echo
+echo "testing LUKS qcow2 encryption"
+echo
+
+_make_test_img --object $SECRET -o 
"encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=64K"
 $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+echo
+echo "testing legacy AES qcow2 encryption"
+echo
+
+
+_make_test_img --object $SECRET -o 
"encrypt.format=aes,encrypt.key-secret=sec0,cluster_size=64K" $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/263.out b/tests/qemu-iotests/263.out
new file mode 100644
index 00..0c982c55cb
--- /dev/null
+++ b/tests/qemu-iotests/263.out
@@ -0,0 +1,40 @@
+QA output created by 263
+
+testing LUKS qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+testing legacy AES qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=aes 
encrypt.key-secret=sec0
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and

[Qemu-devel] [PATCH v2 1/2] blockdev: release the AioContext at drive_backup_prepare

[Sergio Lopez]

do_drive_backup() already acquires the AioContext, so release it
before the call.

Signed-off-by: Sergio Lopez 
---
 blockdev.c | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index fbef6845c8..3927fdab80 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1783,20 +1783,16 @@ static void drive_backup_prepare(BlkActionState 
*common, Error **errp)
 
 aio_context = bdrv_get_aio_context(bs);
 aio_context_acquire(aio_context);
-
 /* Paired with .clean() */
 bdrv_drained_begin(bs);
+aio_context_release(aio_context);
 
 state->bs = bs;
 
 state->job = do_drive_backup(backup, common->block_job_txn, &local_err);
 if (local_err) {
 error_propagate(errp, local_err);
-goto out;
 }
-
-out:
-aio_context_release(aio_context);
 }
 
 static void drive_backup_commit(BlkActionState *common)
-- 
2.21.0

[Qemu-devel] [PATCH v2 0/2] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

[Sergio Lopez]

do_drive_backup() acquires the AioContext lock of the corresponding
BlockDriverState. This is not a problem when it's called from
qmp_drive_backup(), but drive_backup_prepare() also acquires the lock
before calling it.

Additionally, Max Reitz pointed out that bdrv_try_set_aio_context() is
called at do_backup_common() with the new context held, and the old
context not held, while it expects it to be the other way
around. This is also the case for other uses of that function on
blockdev.c.

This patch series fixes all occurrences of bdrv_try_set_aio_context()
to honor the context requirements. It also changes
drive_backup_prepare() to release the context before calling
do_drive_backup().

---
Changelog

v2:
 - Honor bdrv_try_set_aio_context() context acquisition requirements
   (thanks Max Reitz).
 - Release the context at drive_backup_prepare() instead of avoiding
   re-acquiring it at do_drive_baclup(). (thanks Max Reitz)
 - Convert a single patch into a two-patch series.
---

Sergio Lopez (2):
  blockdev: release the AioContext at drive_backup_prepare
  blockdev: honor bdrv_try_set_aio_context() context requirements

 blockdev.c | 127 -
 1 file changed, 76 insertions(+), 51 deletions(-)

-- 
2.21.0

[Qemu-devel] [PATCH v2 2/2] blockdev: honor bdrv_try_set_aio_context() context requirements

[Sergio Lopez]

bdrv_try_set_aio_context() requires that the old context is held, and
the new context is not held. Fix all the ocurrences where it's not
done this way.

Suggested-by: Max Reitz 
Signed-off-by: Sergio Lopez 
---
 blockdev.c | 121 +
 1 file changed, 75 insertions(+), 46 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 3927fdab80..2534fef389 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1536,6 +1536,7 @@ static void external_snapshot_prepare(BlkActionState 
*common,
  DO_UPCAST(ExternalSnapshotState, common, common);
 TransactionAction *action = common->action;
 AioContext *aio_context;
+AioContext *old_context;
 int ret;
 
 /* 'blockdev-snapshot' and 'blockdev-snapshot-sync' have similar
@@ -1575,30 +1576,30 @@ static void external_snapshot_prepare(BlkActionState 
*common,
 
 aio_context = bdrv_get_aio_context(state->old_bs);
 aio_context_acquire(aio_context);
-
 /* Paired with .clean() */
 bdrv_drained_begin(state->old_bs);
+aio_context_release(aio_context);
 
 if (!bdrv_is_inserted(state->old_bs)) {
 error_setg(errp, QERR_DEVICE_HAS_NO_MEDIUM, device);
-goto out;
+return;
 }
 
 if (bdrv_op_is_blocked(state->old_bs,
BLOCK_OP_TYPE_EXTERNAL_SNAPSHOT, errp)) {
-goto out;
+return;
 }
 
 if (!bdrv_is_read_only(state->old_bs)) {
 if (bdrv_flush(state->old_bs)) {
 error_setg(errp, QERR_IO_ERROR);
-goto out;
+return;
 }
 }
 
 if (!bdrv_is_first_non_filter(state->old_bs)) {
 error_setg(errp, QERR_FEATURE_DISABLED, "snapshot");
-goto out;
+return;
 }
 
 if (action->type == TRANSACTION_ACTION_KIND_BLOCKDEV_SNAPSHOT_SYNC) {
@@ -1610,13 +1611,13 @@ static void external_snapshot_prepare(BlkActionState 
*common,
 
 if (node_name && !snapshot_node_name) {
 error_setg(errp, "New overlay node name missing");
-goto out;
+return;
 }
 
 if (snapshot_node_name &&
 bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
 error_setg(errp, "New overlay node name already in use");
-goto out;
+return;
 }
 
 flags = state->old_bs->open_flags;
@@ -1629,7 +1630,7 @@ static void external_snapshot_prepare(BlkActionState 
*common,
 int64_t size = bdrv_getlength(state->old_bs);
 if (size < 0) {
 error_setg_errno(errp, -size, "bdrv_getlength failed");
-goto out;
+return;
 }
 bdrv_refresh_filename(state->old_bs);
 bdrv_img_create(new_image_file, format,
@@ -1638,7 +1639,7 @@ static void external_snapshot_prepare(BlkActionState 
*common,
 NULL, size, flags, false, &local_err);
 if (local_err) {
 error_propagate(errp, local_err);
-goto out;
+return;
 }
 }
 
@@ -1653,34 +1654,41 @@ static void external_snapshot_prepare(BlkActionState 
*common,
   errp);
 /* We will manually add the backing_hd field to the bs later */
 if (!state->new_bs) {
-goto out;
+return;
 }
 
 if (bdrv_has_blk(state->new_bs)) {
 error_setg(errp, "The overlay is already in use");
-goto out;
+return;
 }
 
 if (bdrv_op_is_blocked(state->new_bs, BLOCK_OP_TYPE_EXTERNAL_SNAPSHOT,
errp)) {
-goto out;
+return;
 }
 
 if (state->new_bs->backing != NULL) {
 error_setg(errp, "The overlay already has a backing image");
-goto out;
+return;
 }
 
 if (!state->new_bs->drv->supports_backing) {
 error_setg(errp, "The overlay does not support backing images");
-goto out;
+return;
 }
 
+old_context = bdrv_get_aio_context(state->new_bs);
+aio_context_acquire(old_context);
+
 ret = bdrv_try_set_aio_context(state->new_bs, aio_context, errp);
 if (ret < 0) {
-goto out;
+aio_context_release(old_context);
+return;
 }
 
+aio_context_release(old_context);
+aio_context_acquire(aio_context);
+
 /* This removes our old bs and adds the new bs. This is an operation that
  * can fail, so we need to do it in .prepare; undoing it for abort is
  * always possible. */
@@ -1688,11 +1696,10 @@ static void external_snapshot_prepare(BlkActionState 
*common,
 bdrv_append(state->new_bs, state->old_bs, &local_err);
 if (local_err) {
 error_propagate(errp, local_err);
-goto out;
+} else {
+state->overlay_appended = true;
 }
-state->overlay_appended = true;
 
-out:
 aio_context_release(aio_context);
 }
 
@@ -3490,13 +3497,11 @@ out:
 static BlockJob *do_backup_common(B

Re: [Qemu-devel] [PATCH v8 18/32] riscv: sifive_u: Set the minimum number of cpus to 2

[Bin Meng]

Hi Palmer,

On Fri, Sep 13, 2019 at 10:33 PM Palmer Dabbelt  wrote:
>
> On Fri, 06 Sep 2019 09:20:05 PDT (-0700), bmeng...@gmail.com wrote:
> > It is not useful if we only have one management CPU.
> >
> > Signed-off-by: Bin Meng 
> > Reviewed-by: Alistair Francis 
> >
> > ---
> >
> > Changes in v8: None
> > Changes in v7: None
> > Changes in v6: None
> > Changes in v5: None
> > Changes in v4: None
> > Changes in v3:
> > - use management cpu count + 1 for the min_cpus
> >
> > Changes in v2:
> > - update the file header to indicate at least 2 harts are created
> >
> >  hw/riscv/sifive_u.c | 4 +++-
> >  include/hw/riscv/sifive_u.h | 2 ++
> >  2 files changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/riscv/sifive_u.c b/hw/riscv/sifive_u.c
> > index 2947e06..2023b71 100644
> > --- a/hw/riscv/sifive_u.c
> > +++ b/hw/riscv/sifive_u.c
> > @@ -10,7 +10,8 @@
> >   * 1) CLINT (Core Level Interruptor)
> >   * 2) PLIC (Platform Level Interrupt Controller)
> >   *
> > - * This board currently uses a hardcoded devicetree that indicates one 
> > hart.
> > + * This board currently generates devicetree dynamically that indicates at 
> > least
> > + * two harts.
> >   *
> >   * This program is free software; you can redistribute it and/or modify it
> >   * under the terms and conditions of the GNU General Public License,
> > @@ -433,6 +434,7 @@ static void riscv_sifive_u_machine_init(MachineClass 
> > *mc)
> >   * management CPU.
> >   */
> >  mc->max_cpus = 4;
> > +mc->min_cpus = SIFIVE_U_MANAGEMENT_CPU_COUNT + 1;
> >  }
> >
> >  DEFINE_MACHINE("sifive_u", riscv_sifive_u_machine_init)
> > diff --git a/include/hw/riscv/sifive_u.h b/include/hw/riscv/sifive_u.h
> > index f25bad8..6d22741 100644
> > --- a/include/hw/riscv/sifive_u.h
> > +++ b/include/hw/riscv/sifive_u.h
> > @@ -69,6 +69,8 @@ enum {
> >  SIFIVE_U_GEM_CLOCK_FREQ = 12500
> >  };
> >
> > +#define SIFIVE_U_MANAGEMENT_CPU_COUNT   1
> > +
> >  #define SIFIVE_U_PLIC_HART_CONFIG "MS"
> >  #define SIFIVE_U_PLIC_NUM_SOURCES 54
> >  #define SIFIVE_U_PLIC_NUM_PRIORITIES 7
>
> This fails "make check", so I'm going to squash in this
>
> diff --git a/hw/riscv/sifive_u.c b/hw/riscv/sifive_u.c
> index ca9f7fea41..adecbf1dd9 100644
> --- a/hw/riscv/sifive_u.c
> +++ b/hw/riscv/sifive_u.c
> @@ -528,6 +528,7 @@ static void riscv_sifive_u_machine_init(MachineClass *mc)
>  mc->init = riscv_sifive_u_init;
>  mc->max_cpus = SIFIVE_U_MANAGEMENT_CPU_COUNT + 
> SIFIVE_U_COMPUTE_CPU_COUNT;
>  mc->min_cpus = SIFIVE_U_MANAGEMENT_CPU_COUNT + 1;
> +mc->default_cpus = mc->max_cpus;

Thank you for fixing the 'make check'. Shouldn't it be:

mc->default_cpus = mc->min_cpus;

?

>  }
>
>  DEFINE_MACHINE("sifive_u", riscv_sifive_u_machine_init)

Regards,
Bin

[Qemu-devel] [PATCH v6 5/6] target/arm: remove run-time semihosting checks for linux-user

[Alex Bennée]

Now we do all our checking at translate time we can make cpu_loop a
little bit simpler. We also introduce a simple linux-user semihosting
test case to defend the functionality. The out-of-tree softmmu based
semihosting tests are still more comprehensive.

Signed-off-by: Alex Bennée 
---
 linux-user/arm/cpu_loop.c   | 3 ---
 linux-user/arm/target_syscall.h | 3 ---
 2 files changed, 6 deletions(-)

diff --git a/linux-user/arm/cpu_loop.c b/linux-user/arm/cpu_loop.c
index 8d65de5b9f4..e28c45cd4ab 100644
--- a/linux-user/arm/cpu_loop.c
+++ b/linux-user/arm/cpu_loop.c
@@ -325,9 +325,6 @@ void cpu_loop(CPUARMState *env)
 
 if (n == ARM_NR_cacheflush) {
 /* nop */
-} else if (n == ARM_NR_semihosting
-   || n == ARM_NR_thumb_semihosting) {
-env->regs[0] = do_arm_semihosting (env);
 } else if (n == 0 || n >= ARM_SYSCALL_BASE || env->thumb) {
 /* linux syscall */
 if (env->thumb || n == 0) {
diff --git a/linux-user/arm/target_syscall.h b/linux-user/arm/target_syscall.h
index afc0772e194..f85cbdaf56f 100644
--- a/linux-user/arm/target_syscall.h
+++ b/linux-user/arm/target_syscall.h
@@ -18,9 +18,6 @@ struct target_pt_regs {
 #define ARM_NR_set_tls   (ARM_NR_BASE + 5)
 #define ARM_NR_get_tls(ARM_NR_BASE + 6)
 
-#define ARM_NR_semihosting   0x123456
-#define ARM_NR_thumb_semihosting  0xAB
-
 #if defined(TARGET_WORDS_BIGENDIAN)
 #define UNAME_MACHINE "armv5teb"
 #else
-- 
2.20.1

[Qemu-devel] [PATCH v6 4/6] target/arm: remove run time semihosting checks

[Alex Bennée]

Now we do all our checking and use a common EXCP_SEMIHOST for
semihosting operations we can make helper code a lot simpler.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
---
v2
  - fix re-base conflicts
  - hoist EXCP_SEMIHOST check
  - comment cleanups
v5
  - move CONFIG_TCG ifdefs
---
 target/arm/helper.c | 96 +++--
 1 file changed, 22 insertions(+), 74 deletions(-)

diff --git a/target/arm/helper.c b/target/arm/helper.c
index 507026c9154..a87ae6d46a1 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -8339,88 +8339,32 @@ static void arm_cpu_do_interrupt_aarch64(CPUState *cs)
   new_el, env->pc, pstate_read(env));
 }
 
-static inline bool check_for_semihosting(CPUState *cs)
-{
+/*
+ * Do semihosting call and set the appropriate return value. All the
+ * permission and validity checks have been done at translate time.
+ *
+ * We only see semihosting exceptions in TCG only as they are not
+ * trapped to the hypervisor in KVM.
+ */
 #ifdef CONFIG_TCG
-/* Check whether this exception is a semihosting call; if so
- * then handle it and return true; otherwise return false.
- */
+static void handle_semihosting(CPUState *cs)
+{
 ARMCPU *cpu = ARM_CPU(cs);
 CPUARMState *env = &cpu->env;
 
 if (is_a64(env)) {
-if (cs->exception_index == EXCP_SEMIHOST) {
-/* This is always the 64-bit semihosting exception.
- * The "is this usermode" and "is semihosting enabled"
- * checks have been done at translate time.
- */
-qemu_log_mask(CPU_LOG_INT,
-  "...handling as semihosting call 0x%" PRIx64 "\n",
-  env->xregs[0]);
-env->xregs[0] = do_arm_semihosting(env);
-return true;
-}
-return false;
+qemu_log_mask(CPU_LOG_INT,
+  "...handling as semihosting call 0x%" PRIx64 "\n",
+  env->xregs[0]);
+env->xregs[0] = do_arm_semihosting(env);
 } else {
-uint32_t imm;
-
-/* Only intercept calls from privileged modes, to provide some
- * semblance of security.
- */
-if (cs->exception_index != EXCP_SEMIHOST &&
-(!semihosting_enabled() ||
- ((env->uncached_cpsr & CPSR_M) == ARM_CPU_MODE_USR))) {
-return false;
-}
-
-switch (cs->exception_index) {
-case EXCP_SEMIHOST:
-/* This is always a semihosting call; the "is this usermode"
- * and "is semihosting enabled" checks have been done at
- * translate time.
- */
-break;
-case EXCP_SWI:
-/* Check for semihosting interrupt.  */
-if (env->thumb) {
-imm = arm_lduw_code(env, env->regs[15] - 2, arm_sctlr_b(env))
-& 0xff;
-if (imm == 0xab) {
-break;
-}
-} else {
-imm = arm_ldl_code(env, env->regs[15] - 4, arm_sctlr_b(env))
-& 0xff;
-if (imm == 0x123456) {
-break;
-}
-}
-return false;
-case EXCP_BKPT:
-/* See if this is a semihosting syscall.  */
-if (env->thumb) {
-imm = arm_lduw_code(env, env->regs[15], arm_sctlr_b(env))
-& 0xff;
-if (imm == 0xab) {
-env->regs[15] += 2;
-break;
-}
-}
-return false;
-default:
-return false;
-}
-
 qemu_log_mask(CPU_LOG_INT,
   "...handling as semihosting call 0x%x\n",
   env->regs[0]);
 env->regs[0] = do_arm_semihosting(env);
-return true;
 }
-#else
-return false;
-#endif
 }
+#endif
 
 /* Handle a CPU exception for A and R profile CPUs.
  * Do any appropriate logging, handle PSCI calls, and then hand off
@@ -8451,13 +8395,17 @@ void arm_cpu_do_interrupt(CPUState *cs)
 return;
 }
 
-/* Semihosting semantics depend on the register width of the
- * code that caused the exception, not the target exception level,
- * so must be handled here.
+/*
+ * Semihosting semantics depend on the register width of the code
+ * that caused the exception, not the target exception level, so
+ * must be handled here.
  */
-if (check_for_semihosting(cs)) {
+#ifdef CONFIG_TCG
+if (cs->exception_index == EXCP_SEMIHOST) {
+handle_semihosting(cs);
 return;
 }
+#endif
 
 /* Hooks may change global state so BQL should be held, also the
  * BQL needs to be held for any modification of
-- 
2.20.1

[Qemu-devel] [PATCH v6 2/6] target/arm: handle M-profile semihosting at translate time

[Alex Bennée]

We do this for other semihosting calls so we might as well do it for
M-profile as well.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 

---
v2
  - update for change to gen_exception_internal_insn API
v3
  - update for decode tree
v4
  - use !IS_USER
v5
  - return #ifndef CONFIG_USER
---
 target/arm/m_helper.c  | 18 ++
 target/arm/translate.c | 11 ++-
 2 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/target/arm/m_helper.c b/target/arm/m_helper.c
index 884d35d2b02..27cd2f3f964 100644
--- a/target/arm/m_helper.c
+++ b/target/arm/m_helper.c
@@ -2114,19 +2114,13 @@ void arm_v7m_cpu_do_interrupt(CPUState *cs)
 break;
 }
 break;
+case EXCP_SEMIHOST:
+qemu_log_mask(CPU_LOG_INT,
+  "...handling as semihosting call 0x%x\n",
+  env->regs[0]);
+env->regs[0] = do_arm_semihosting(env);
+return;
 case EXCP_BKPT:
-if (semihosting_enabled()) {
-int nr;
-nr = arm_lduw_code(env, env->regs[15], arm_sctlr_b(env)) & 0xff;
-if (nr == 0xab) {
-env->regs[15] += 2;
-qemu_log_mask(CPU_LOG_INT,
-  "...handling as semihosting call 0x%x\n",
-  env->regs[0]);
-env->regs[0] = do_arm_semihosting(env);
-return;
-}
-}
 armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_DEBUG, false);
 break;
 case EXCP_IRQ:
diff --git a/target/arm/translate.c b/target/arm/translate.c
index 34bb280e3da..b5272119330 100644
--- a/target/arm/translate.c
+++ b/target/arm/translate.c
@@ -8424,7 +8424,16 @@ static bool trans_BKPT(DisasContext *s, arg_BKPT *a)
 if (!ENABLE_ARCH_5) {
 return false;
 }
-gen_exception_bkpt_insn(s, syn_aa32_bkpt(a->imm, false));
+if (arm_dc_feature(s, ARM_FEATURE_M) &&
+semihosting_enabled() &&
+#ifndef CONFIG_USER_ONLY
+!IS_USER(s) &&
+#endif
+(a->imm == 0xab)) {
+gen_exception_internal_insn(s, s->base.pc_next, EXCP_SEMIHOST);
+} else {
+gen_exception_bkpt_insn(s, syn_aa32_bkpt(a->imm, false));
+}
 return true;
 }
 
-- 
2.20.1