Re: [Qemu-devel] [PULL v2 0/7] Block patches for 4.1.0-rc4

[Peter Maydell]

On Tue, 6 Aug 2019 at 12:59, Max Reitz  wrote:
>
> The following changes since commit 9bb68d34dda9be60335e73e65c8fb61bca035362:
>
>   Merge remote-tracking branch 
> 'remotes/philmd-gitlab/tags/edk2-next-20190803' into staging (2019-08-05 
> 11:05:36 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/XanClic/qemu.git tags/pull-block-2019-08-06
>
> for you to fetch changes up to 110571be4e70ac015628e76d2731f96dd8d1998c:
>
>   block/backup: disable copy_range for compressed backup (2019-08-06 13:17:27 
> +0200)
> 
> v2: Added “Cc: qemu-stable” tag where necessary
>
> 
> Block patches for 4.1.0-rc4:
> - Fix the backup block job when using copy offloading
> - Fix the mirror block job when using the write-blocking copy mode
> - Fix incremental backups after the image has been grown with the
>   respective bitmap attached to it
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.1
for any user-visible changes.

-- PMM

[Qemu-devel] [PULL v2 0/7] Block patches for 4.1.0-rc4

[Max Reitz]

The following changes since commit 9bb68d34dda9be60335e73e65c8fb61bca035362:

  Merge remote-tracking branch 'remotes/philmd-gitlab/tags/edk2-next-20190803' 
into staging (2019-08-05 11:05:36 +0100)

are available in the Git repository at:

  https://github.com/XanClic/qemu.git tags/pull-block-2019-08-06

for you to fetch changes up to 110571be4e70ac015628e76d2731f96dd8d1998c:

  block/backup: disable copy_range for compressed backup (2019-08-06 13:17:27 
+0200)

v2: Added “Cc: qemu-stable” tag where necessary


Block patches for 4.1.0-rc4:
- Fix the backup block job when using copy offloading
- Fix the mirror block job when using the write-blocking copy mode
- Fix incremental backups after the image has been grown with the
  respective bitmap attached to it


Max Reitz (5):
  backup: Copy only dirty areas
  iotests: Test backup job with two guest writes
  iotests: Test incremental backup after truncation
  mirror: Only mirror granularity-aligned chunks
  iotests: Test unaligned blocking mirror write

Vladimir Sementsov-Ogievskiy (2):
  util/hbitmap: update orig_size on truncate
  block/backup: disable copy_range for compressed backup

 block/backup.c | 15 ---
 block/mirror.c | 29 
 util/hbitmap.c |  6 +-
 tests/qemu-iotests/056 | 39 ++
 tests/qemu-iotests/056.out |  4 ++--
 tests/qemu-iotests/124 | 38 +
 tests/qemu-iotests/124.out |  4 ++--
 tests/qemu-iotests/151 | 25 
 tests/qemu-iotests/151.out |  4 ++--
 9 files changed, 150 insertions(+), 14 deletions(-)

-- 
2.21.0

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 11 November 2015 at 20:59, Marc-André Lureau
 wrote:
> Hi Peter
>
> On Wed, Nov 11, 2015 at 9:33 PM, Peter Maydell  
> wrote:
>> On 10 November 2015 at 18:41, Peter Maydell  wrote:
>>> On 9 November 2015 at 17:50, Marc-André Lureau  wrote:
 I can imagine a test starting a server thread and 2 qemu instances
 would take more than 5s on such configuration then.

 Could you try timing the test a few times to confirm this?
>>>
>>> petmay01@moonshot-dsg-11:~/qemu/build/all-a64$ time
>>> QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386
>>> QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM %
>>> 255 + 1))} gtester -k --verbose -m=quick tests/ivshmem-test
>>> TEST: tests/ivshmem-test... (pid=10893)
>>>   /i386/ivshmem/single:OK
>>>   /i386/ivshmem/pair:  OK
>>>   /i386/ivshmem/server:OK
>>>   /i386/ivshmem/hotplug:   OK
>>>   /i386/ivshmem/memdev:OK
>>> PASS: tests/ivshmem-test
>>>
>>> real0m11.945s
>>> user0m11.020s
>>> sys 0m0.310s
>>>
>>> (almost all of the runtime seems to be in the "pair" subtest).
>>
>> This is now failing on practically every pull request I test.
>> Please post a patch to fix this test or disable it...
>
> This is the simplest patch I suggest for now.

That will still mean that trying the slow tests gives random
failures, so this still needs attention (ie raising the timeouts
to something that won't actually be hit), but I guess it will
solve my immediate problem. Can you send it to the list as
a proper patch, please?

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 10 November 2015 at 18:41, Peter Maydell  wrote:
> On 9 November 2015 at 17:50, Marc-André Lureau  wrote:
>> I can imagine a test starting a server thread and 2 qemu instances
>> would take more than 5s on such configuration then.
>>
>> Could you try timing the test a few times to confirm this?
>
> petmay01@moonshot-dsg-11:~/qemu/build/all-a64$ time
> QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386
> QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM %
> 255 + 1))} gtester -k --verbose -m=quick tests/ivshmem-test
> TEST: tests/ivshmem-test... (pid=10893)
>   /i386/ivshmem/single:OK
>   /i386/ivshmem/pair:  OK
>   /i386/ivshmem/server:OK
>   /i386/ivshmem/hotplug:   OK
>   /i386/ivshmem/memdev:OK
> PASS: tests/ivshmem-test
>
> real0m11.945s
> user0m11.020s
> sys 0m0.310s
>
> (almost all of the runtime seems to be in the "pair" subtest).

This is now failing on practically every pull request I test.
Please post a patch to fix this test or disable it...

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Marc-André Lureau]

Hi Peter

On Wed, Nov 11, 2015 at 9:33 PM, Peter Maydell  wrote:
> On 10 November 2015 at 18:41, Peter Maydell  wrote:
>> On 9 November 2015 at 17:50, Marc-André Lureau  wrote:
>>> I can imagine a test starting a server thread and 2 qemu instances
>>> would take more than 5s on such configuration then.
>>>
>>> Could you try timing the test a few times to confirm this?
>>
>> petmay01@moonshot-dsg-11:~/qemu/build/all-a64$ time
>> QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386
>> QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM %
>> 255 + 1))} gtester -k --verbose -m=quick tests/ivshmem-test
>> TEST: tests/ivshmem-test... (pid=10893)
>>   /i386/ivshmem/single:OK
>>   /i386/ivshmem/pair:  OK
>>   /i386/ivshmem/server:OK
>>   /i386/ivshmem/hotplug:   OK
>>   /i386/ivshmem/memdev:OK
>> PASS: tests/ivshmem-test
>>
>> real0m11.945s
>> user0m11.020s
>> sys 0m0.310s
>>
>> (almost all of the runtime seems to be in the "pair" subtest).
>
> This is now failing on practically every pull request I test.
> Please post a patch to fix this test or disable it...

This is the simplest patch I suggest for now.

-- 
Marc-André Lureau
From 44f229675d4c9ea896fa83e8cbe5add4c5846541 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= 
Date: Wed, 11 Nov 2015 21:47:05 +0100
Subject: [PATCH] tests: classify some ivshmem tests as slow
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Some tests may take long to run, move them under g_test_slow()
condition.

Signed-off-by: Marc-André Lureau 
---
 tests/ivshmem-test.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/ivshmem-test.c b/tests/ivshmem-test.c
index c8f0cf0..f1793ba 100644
--- a/tests/ivshmem-test.c
+++ b/tests/ivshmem-test.c
@@ -478,10 +478,12 @@ int main(int argc, char **argv)
 tmpserver = g_strconcat(tmpdir, "/server", NULL);
 
 qtest_add_func("/ivshmem/single", test_ivshmem_single);
-qtest_add_func("/ivshmem/pair", test_ivshmem_pair);
-qtest_add_func("/ivshmem/server", test_ivshmem_server);
 qtest_add_func("/ivshmem/hotplug", test_ivshmem_hotplug);
 qtest_add_func("/ivshmem/memdev", test_ivshmem_memdev);
+if (g_test_slow()) {
+qtest_add_func("/ivshmem/pair", test_ivshmem_pair);
+qtest_add_func("/ivshmem/server", test_ivshmem_server);
+}
 
 ret = g_test_run();
 
-- 
2.5.0

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 9 November 2015 at 17:50, Marc-André Lureau  wrote:
> Hi
>
> - Original Message -
>> On 9 November 2015 at 15:09, Marc-André Lureau  wrote:
>> >> On 9 November 2015 at 12:51, Peter Maydell 
>> >> wrote:
>> >> Marc-André, can you look into why the ivshmem tests might be
>> >> intermittently
>> >> failing like this, please?
>> >
>> > Is this with an slow or emulated host? It could be that the 5s timeout
>> > is not enough?
>>
>> This is with the 32-bit build on a 64-bit ARM server box. So it's
>> not the fastest machine in the world, but it's not bad either.
>> It will be using TCG, obviously.
>>
>> A test which takes 5 seconds to run isn't ideal from a "keep
>> the make-check time down" perspective either.
>
> I can imagine a test starting a server thread and 2 qemu instances would take 
> more than 5s on such configuration then.
>
> Could you try timing the test a few times to confirm this?

petmay01@moonshot-dsg-11:~/qemu/build/all-a64$ time
QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386
QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM %
255 + 1))} gtester -k --verbose -m=quick tests/ivshmem-test
TEST: tests/ivshmem-test... (pid=10893)
  /i386/ivshmem/single:OK
  /i386/ivshmem/pair:  OK
  /i386/ivshmem/server:OK
  /i386/ivshmem/hotplug:   OK
  /i386/ivshmem/memdev:OK
PASS: tests/ivshmem-test

real0m11.945s
user0m11.020s
sys 0m0.310s

(almost all of the runtime seems to be in the "pair" subtest).

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 9 November 2015 at 12:51, Peter Maydell  wrote:
> On 9 November 2015 at 10:08, Stefan Hajnoczi  wrote:
>> The following changes since commit c4a7bf54e588ff2de9fba0898b686156251b2063:
>>
>>   Merge remote-tracking branch 'remotes/jnsnow/tags/ide-pull-request' into 
>> staging (2015-11-07 19:55:15 +)
>>
>> are available in the git repository at:
>>
>>   git://github.com/stefanha/qemu.git tags/block-pull-request
>>
>> for you to fetch changes up to 84aa0140dd4f04f5d993f0db14c40da8d3de2706:
>>
>>   blockdev: acquire AioContext in hmp_commit() (2015-11-09 10:07:10 +)
>>
>> 
>>
>> 
>
> I got that intermittent ivshmem test failure again :-(
>
> ERROR:/home/petmay01/qemu/tests/ivshmem-test.c:345:test_ivshmem_server:
> assertion failed (ret != 0): (0 != 0)
>
> I'll retry the tests...

Yep, worked fine second time, so applied.

Marc-André, can you look into why the ivshmem tests might be intermittently
failing like this, please?

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 9 November 2015 at 10:08, Stefan Hajnoczi  wrote:
> The following changes since commit c4a7bf54e588ff2de9fba0898b686156251b2063:
>
>   Merge remote-tracking branch 'remotes/jnsnow/tags/ide-pull-request' into 
> staging (2015-11-07 19:55:15 +)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 84aa0140dd4f04f5d993f0db14c40da8d3de2706:
>
>   blockdev: acquire AioContext in hmp_commit() (2015-11-09 10:07:10 +)
>
> 
>
> 

I got that intermittent ivshmem test failure again :-(

ERROR:/home/petmay01/qemu/tests/ivshmem-test.c:345:test_ivshmem_server:
assertion failed (ret != 0): (0 != 0)

I'll retry the tests...

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Marc-André Lureau]

Hi

- Original Message -
> On 9 November 2015 at 12:51, Peter Maydell  wrote:
> > On 9 November 2015 at 10:08, Stefan Hajnoczi  wrote:
> >> The following changes since commit
> >> c4a7bf54e588ff2de9fba0898b686156251b2063:
> >>
> >>   Merge remote-tracking branch 'remotes/jnsnow/tags/ide-pull-request' into
> >>   staging (2015-11-07 19:55:15 +)
> >>
> >> are available in the git repository at:
> >>
> >>   git://github.com/stefanha/qemu.git tags/block-pull-request
> >>
> >> for you to fetch changes up to 84aa0140dd4f04f5d993f0db14c40da8d3de2706:
> >>
> >>   blockdev: acquire AioContext in hmp_commit() (2015-11-09 10:07:10 +)
> >>
> >> 
> >>
> >> 
> >
> > I got that intermittent ivshmem test failure again :-(
> >
> > ERROR:/home/petmay01/qemu/tests/ivshmem-test.c:345:test_ivshmem_server:
> > assertion failed (ret != 0): (0 != 0)
> >
> > I'll retry the tests...
> 
> Yep, worked fine second time, so applied.
> 
> Marc-André, can you look into why the ivshmem tests might be intermittently
> failing like this, please?

Is this with an slow or emulated host? It could be that the 5s timeout is not 
enough?

Alternatively, I think I could come up with a framework to put some kind of 
breakpoints in qemu, and wait until they happen. This would get rid of the 
timer.

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Marc-André Lureau]

Hi

- Original Message -
> On 9 November 2015 at 15:09, Marc-André Lureau  wrote:
> >> On 9 November 2015 at 12:51, Peter Maydell 
> >> wrote:
> >> Marc-André, can you look into why the ivshmem tests might be
> >> intermittently
> >> failing like this, please?
> >
> > Is this with an slow or emulated host? It could be that the 5s timeout
> > is not enough?
> 
> This is with the 32-bit build on a 64-bit ARM server box. So it's
> not the fastest machine in the world, but it's not bad either.
> It will be using TCG, obviously.
> 
> A test which takes 5 seconds to run isn't ideal from a "keep
> the make-check time down" perspective either.

I can imagine a test starting a server thread and 2 qemu instances would take 
more than 5s on such configuration then.

Could you try timing the test a few times to confirm this?

If it's too long, I suggest we move it to g_test_slow(), and perhaps get rid of 
the timer. 

thanks

Re: [Qemu-devel] [PULL v2 0/7] Block patches

[Peter Maydell]

On 9 November 2015 at 15:09, Marc-André Lureau  wrote:
>> On 9 November 2015 at 12:51, Peter Maydell  wrote:
>> Marc-André, can you look into why the ivshmem tests might be intermittently
>> failing like this, please?
>
> Is this with an slow or emulated host? It could be that the 5s timeout
> is not enough?

This is with the 32-bit build on a 64-bit ARM server box. So it's
not the fastest machine in the world, but it's not bad either.
It will be using TCG, obviously.

A test which takes 5 seconds to run isn't ideal from a "keep
the make-check time down" perspective either.

thanks
-- PMM

[Qemu-devel] [PULL v2 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit c4a7bf54e588ff2de9fba0898b686156251b2063:

  Merge remote-tracking branch 'remotes/jnsnow/tags/ide-pull-request' into 
staging (2015-11-07 19:55:15 +)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 84aa0140dd4f04f5d993f0db14c40da8d3de2706:

  blockdev: acquire AioContext in hmp_commit() (2015-11-09 10:07:10 +)





Denis V. Lunev (1):
  monitor: add missed aio_context_acquire into vm_completion call

Fam Zheng (3):
  aio: Introduce aio_external_disabled
  aio: Introduce aio_context_setup
  aio: Introduce aio-epoll.c

Michael S. Tsirkin (2):
  dataplane: simplify indirect descriptor read
  dataplane: support non-contigious s/g

Stefan Hajnoczi (1):
  blockdev: acquire AioContext in hmp_commit()

 aio-posix.c | 188 +++-
 aio-win32.c |   4 +
 async.c |  13 ++-
 blockdev.c  |  12 ++-
 hw/virtio/dataplane/vring.c |  96 ++
 include/block/aio.h |  24 ++
 monitor.c   |  11 ++-
 7 files changed, 309 insertions(+), 39 deletions(-)

-- 
2.5.0

[Qemu-devel] [PULL v2 0/7] Block patches

[Kevin Wolf]

The following changes since commit d05ef160453e98546a4197496dc8a3cb2defac53:

  Allow clock_gettime() monotonic clock to be utilized on more OS's (2013-04-04 
20:22:45 -0500)

are available in the git repository at:

  git://repo.or.cz/qemu/kevin.git for-anthony

for you to fetch changes up to c2b6ff51e4a3ad1f7ec5dbc94970e9778b31d718:

  qcow2: Fix L1 write error handling in qcow2_update_snapshot_refcount 
(2013-04-05 18:58:05 +0200)


Kevin Wolf (3):
  usb-storage: Forward serial number to scsi-disk
  qcow2: Return real error in qcow2_update_snapshot_refcount
  qcow2: Fix L1 write error handling in qcow2_update_snapshot_refcount

Stefan Hajnoczi (4):
  block: fix I/O throttling accounting blind spot
  block: keep I/O throttling slice time constant
  block: drop duplicated slice extension code
  block: clean up I/O throttling wait_time code

 block.c   | 49 +--
 block/qcow2-refcount.c| 25 
 blockdev.c|  1 -
 hw/pci/pci-hotplug.c  |  2 +-
 hw/scsi-bus.c |  8 ++--
 hw/scsi.h |  3 ++-
 hw/usb/dev-storage.c  |  2 +-
 include/block/block_int.h |  3 +--
 8 files changed, 46 insertions(+), 47 deletions(-)