[PULL v2] Block layer patches
The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:
Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request'
into staging (2020-09-06 16:23:55 +0100)
are available in the Git repository at:
git://repo.or.cz/qemu/kevin.git tags/for-upstream
for you to fetch changes up to c984095a47c30e0952d34e77decf9f4c0f8d5a19:
block/nvme: Pair doorbell registers (2020-09-08 13:40:53 +0200)
Block layer patches:
- qemu-img create: Fail gracefully when backing file is an empty string
- Fixes related to filter block nodes ("Deal with filters" series)
- block/nvme: Various cleanups required to use multiple queues
- block/nvme: Use NvmeBar structure from "block/nvme.h"
- file-win32: Fix "locking" option
- iotests: Allow running from different directory
Connor Kuehl (1):
block: Raise an error when backing file parameter is an empty string
Kevin Wolf (2):
iotests: Allow running from different directory
file-win32: Fix "locking" option
Max Reitz (43):
block: Add child access functions
block: Add chain helper functions
block: bdrv_cow_child() for bdrv_has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
throttle: Support compressed writes
copy-on-read: Support compressed writes
block: Use bdrv_filter_(bs|child) where obvious
block: Use CAFs in block status functions
stream: Deal with filters
block: Use CAFs when working with backing chains
block: Use bdrv_cow_child() in bdrv_co_truncate()
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
vmdk: Drop vmdk_co_flush()
block: Iterate over children in refresh_limits
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate()
block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Improve get_allocated_file_size's default
block/null: Implement bdrv_get_allocated_file_size
blockdev: Use CAF in external_snapshot_prepare()
block: Report data child for query-blockstats
block: Use child access functions for QAPI queries
block-copy: Use CAF to find sync=top base
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Leave BDS.backing_{file,format} constant
iotests: Test that qcow2's data-file is flushed
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing
Philippe Mathieu-Daudé (18):
block/nvme: Replace magic value by SCALE_MS definition
block/nvme: Avoid further processing if trace event not enabled
block/nvme: Let nvme_create_queue_pair() fail gracefully
block/nvme: Define INDEX macros to ease code review
block/nvme: Improve error message when IO queue creation failed
block/nvme: Use common error path in nvme_add_io_queue()
block/nvme: Rename local variable
block/nvme: Use union of NvmeIdCtrl / NvmeIdNs structures
block/nvme: Replace qemu_try_blockalign0 by qemu_try_blockalign/memset
block/nvme: Replace qemu_try_blockalign(bs) by qemu_try_memalign(pg_sz)
block/nvme: Simplify nvme_init_queue() arguments
block/nvme: Replace BDRV_POLL_WHILE by AIO_WAIT_WHILE
block/nvme: Simplify nvme_create_queue_pair() arguments
block/nvme: Extract nvme_poll_queue()
block/nvme: Use an array of EventNotifier
block/nvme: Group controller registers in NVMeRegs structure
block/nvme: Use generic NvmeBar structure
block/nvme: Pair doorbell registers
qapi/block-core.json | 58 +++--
include/block/block.h | 2 +-
include/block/block_int.h | 95 +---
block.c| 492 +
block/backup-top.c | 4 +-
block/backup.c | 9 +-
block/blkdebug.c | 7 +-
block/blklogwrites.c | 1 -
block/block-backend.c | 7 +-
block/block-copy.c | 4 +-
block/commit.c | 95 +---
block/copy-on-read.c | 13 +-
block/file-win32.c | 22 +-
block/filter-compress.c| 2 -
block/io.c | 142 ++--
block/mirror.c | 119 +++-
[PULL v2] Block layer patches
The following changes since commit 8796c64ecdfd34be394ea277a53df0c76996: Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1: qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200) Block layer patches: - file-posix: Mitigate file fragmentation with extent size hints - Tighten qemu-img rules on missing backing format - qemu-img map: Don't limit block status request size Eric Blake (10): qemu-img: Flush stdout before before potential stderr messages block: Finish deprecation of 'qemu-img convert -n -o' sheepdog: Add trivial backing_fmt support vmdk: Add trivial backing_fmt support qcow: Tolerate backing_fmt= block: Error if backing file fails during creation without -u qcow2: Deprecate use of qemu-img amend to change backing file iotests: Specify explicit backing format where sensible block: Add support to warn on backing file change without format qemu-img: Deprecate use of -b without -F Kevin Wolf (2): qemu-img map: Don't limit block status request size file-posix: Mitigate file fragmentation with extent size hints Max Reitz (1): iotests: Simplify _filter_img_create() a bit qapi/block-core.json | 11 +++-- docs/system/deprecated.rst | 58 ++ docs/tools/qemu-img.rst | 4 ++ include/block/block.h| 4 +- include/block/block_int.h| 1 + block.c | 53 +--- block/file-posix.c | 42 +++ block/qcow.c | 20 - block/qcow2.c| 7 +++- block/sheepdog.c | 18 +++- block/stream.c | 2 +- block/vmdk.c | 14 +++ blockdev.c | 3 +- qemu-img.c | 20 + tests/qemu-iotests/017 | 2 +- tests/qemu-iotests/017.out | 2 +- tests/qemu-iotests/018 | 2 +- tests/qemu-iotests/018.out | 2 +- tests/qemu-iotests/019 | 5 ++- tests/qemu-iotests/019.out | 2 +- tests/qemu-iotests/020 | 31 -- tests/qemu-iotests/020.out | 15 +-- tests/qemu-iotests/024 | 8 ++-- tests/qemu-iotests/024.out | 5 ++- tests/qemu-iotests/028 | 4 +- tests/qemu-iotests/028.out | 2 +- tests/qemu-iotests/030 | 26 tests/qemu-iotests/034 | 2 +- tests/qemu-iotests/034.out | 2 +- tests/qemu-iotests/037 | 2 +- tests/qemu-iotests/037.out | 2 +- tests/qemu-iotests/038 | 2 +- tests/qemu-iotests/038.out | 2 +- tests/qemu-iotests/039 | 3 +- tests/qemu-iotests/039.out | 2 +- tests/qemu-iotests/040 | 47 +++-- tests/qemu-iotests/041 | 37 +++-- tests/qemu-iotests/042 | 4 +- tests/qemu-iotests/043 | 18 tests/qemu-iotests/043.out | 16 +--- tests/qemu-iotests/046 | 2 +- tests/qemu-iotests/046.out | 2 +- tests/qemu-iotests/049.out | 8 ++-- tests/qemu-iotests/050 | 4 +- tests/qemu-iotests/050.out | 2 +- tests/qemu-iotests/051 | 2 +- tests/qemu-iotests/051.out | 2 +- tests/qemu-iotests/051.pc.out| 2 +- tests/qemu-iotests/054.out | 2 +- tests/qemu-iotests/056 | 3 +- tests/qemu-iotests/060 | 2 +- tests/qemu-iotests/060.out | 2 +- tests/qemu-iotests/061 | 10 ++--- tests/qemu-iotests/061.out | 11 ++--- tests/qemu-iotests/069 | 2 +- tests/qemu-iotests/069.out | 2 +- tests/qemu-iotests/073 | 2 +- tests/qemu-iotests/073.out | 2 +- tests/qemu-iotests/079.out | 2 +- tests/qemu-iotests/082 | 10 +++-- tests/qemu-iotests/082.out | 30 +++--- tests/qemu-iotests/085 | 4 +- tests/qemu-iotests/085.out | 6 +-- tests/qemu-iotests/089 | 2 +- tests/qemu-iotests/089.out | 2 +- tests/qemu-iotests/095 | 4 +- tests/qemu-iotests/095.out | 4 +- tests/qemu-iotests/097 | 4 +- tests/qemu-iotests/097.out | 16 tests/qemu-iotests/098 | 2 +- tests/qemu-iotests/098.out | 8 ++-- tests/qemu-iotests/110 | 4 +- tests/qemu-iotests/110.out | 4 +- tests/qemu-iotests/111.out | 2 +- tests/qemu-iotests/112.out | 4 +- tests/qemu-iotests/114 | 12 ++ tests/qemu-iotests/114.out |
Re: [PULL v2] Block layer patches
On Tue, 8 Sep 2020 at 12:53, Kevin Wolf wrote:
>
> The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:
>
> Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request'
> into staging (2020-09-06 16:23:55 +0100)
>
> are available in the Git repository at:
>
> git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to c984095a47c30e0952d34e77decf9f4c0f8d5a19:
>
> block/nvme: Pair doorbell registers (2020-09-08 13:40:53 +0200)
>
>
> Block layer patches:
>
> - qemu-img create: Fail gracefully when backing file is an empty string
> - Fixes related to filter block nodes ("Deal with filters" series)
> - block/nvme: Various cleanups required to use multiple queues
> - block/nvme: Use NvmeBar structure from "block/nvme.h"
> - file-win32: Fix "locking" option
> - iotests: Allow running from different directory
This fails 'make check' on NetBSD and OpenBSD:
./check: line 47: realpath: command not found
./check: line 60: /common.env: No such file or directory
check: failed to source common.env (make sure the qemu-iotests are run
from tests/qemu-iotests in the build tree)
gmake: *** [/home/qemu/qemu-test.vcb7nz/src/tests/Makefile.include:144:
check-block] Error 1
thanks
-- PMM
Re: [PULL v2] Block layer patches
On 9/9/20 4:55 PM, Peter Maydell wrote: This fails 'make check' on NetBSD and OpenBSD: ./check: line 47: realpath: command not found ./check: line 60: /common.env: No such file or directory check: failed to source common.env (make sure the qemu-iotests are run from tests/qemu-iotests in the build tree) gmake: *** [/home/qemu/qemu-test.vcb7nz/src/tests/Makefile.include:144: check-block] Error 1 BSD has 'readlink -f' (and so does coreutils on Linux), which does the same thing as the Linux-only realpath. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PULL v2] Block layer patches
Am 10.09.2020 um 00:09 hat Eric Blake geschrieben: > On 9/9/20 4:55 PM, Peter Maydell wrote: > > > > > This fails 'make check' on NetBSD and OpenBSD: > > > > ./check: line 47: realpath: command not found > > ./check: line 60: /common.env: No such file or directory > > check: failed to source common.env (make sure the qemu-iotests are run > > from tests/qemu-iotests in the build tree) > > gmake: *** [/home/qemu/qemu-test.vcb7nz/src/tests/Makefile.include:144: > > check-block] Error 1 > > BSD has 'readlink -f' (and so does coreutils on Linux), which does the same > thing as the Linux-only realpath. Thanks, I'll use that in v3. Max, if v3 doesn't go through either, please take over the pull request as I will be on PTO until the 22th. Kevin
Re: [PULL v2] Block layer patches
On 10.09.20 11:13, Kevin Wolf wrote: > Am 10.09.2020 um 00:09 hat Eric Blake geschrieben: >> On 9/9/20 4:55 PM, Peter Maydell wrote: >> >>> >>> This fails 'make check' on NetBSD and OpenBSD: >>> >>> ./check: line 47: realpath: command not found >>> ./check: line 60: /common.env: No such file or directory >>> check: failed to source common.env (make sure the qemu-iotests are run >>> from tests/qemu-iotests in the build tree) >>> gmake: *** [/home/qemu/qemu-test.vcb7nz/src/tests/Makefile.include:144: >>> check-block] Error 1 >> >> BSD has 'readlink -f' (and so does coreutils on Linux), which does the same >> thing as the Linux-only realpath. > > Thanks, I'll use that in v3. > > Max, if v3 doesn't go through either, please take over the pull request > as I will be on PTO until the 22th. OK, sure. (Enjoy your time off. :)) Max signature.asc Description: OpenPGP digital signature
Re: [PULL v2] Block layer patches
On 10/09/2020 00.09, Eric Blake wrote: > On 9/9/20 4:55 PM, Peter Maydell wrote: > >> >> This fails 'make check' on NetBSD and OpenBSD: >> >> ./check: line 47: realpath: command not found >> ./check: line 60: /common.env: No such file or directory >> check: failed to source common.env (make sure the qemu-iotests are run >> from tests/qemu-iotests in the build tree) >> gmake: *** [/home/qemu/qemu-test.vcb7nz/src/tests/Makefile.include:144: >> check-block] Error 1 > > BSD has 'readlink -f' (and so does coreutils on Linux), which does the > same thing as the Linux-only realpath. Seems like readlink -f does not work on macOS: https://cirrus-ci.com/task/5735398972325888?command=main#L7038 Any ideas what to use instead? Peter, why did this slip through your merge tests, do you still skip the iotests there? Thomas
Re: [PULL v2] Block layer patches
On Sat, 12 Sep 2020 at 13:27, Thomas Huth wrote: > Peter, why did this slip through your merge tests, do you still skip the > iotests there? I forget what the reason for them being skipped is, maybe it's because they demand a gnu sed ? thanks -- PMM
Re: [PULL v2] Block layer patches
On 12/09/2020 20.38, Peter Maydell wrote: > On Sat, 12 Sep 2020 at 13:27, Thomas Huth wrote: >> Peter, why did this slip through your merge tests, do you still skip the >> iotests there? > > I forget what the reason for them being skipped is, maybe > it's because they demand a gnu sed ? The tests/check-block.sh script should tell you when you run "make check-block" ... but yes, they need gnu sed - which should be available via homebrew. Thomas
Re: [PULL v2] Block layer patches
On Thu, 9 Jul 2020 at 13:17, Kevin Wolf wrote: > > The following changes since commit 8796c64ecdfd34be394ea277a53df0c76996: > > Merge remote-tracking branch > 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 > 16:33:59 +0100) > > are available in the Git repository at: > > git://repo.or.cz/qemu/kevin.git tags/for-upstream > > for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1: > > qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200) > > > Block layer patches: > > - file-posix: Mitigate file fragmentation with extent size hints > - Tighten qemu-img rules on missing backing format > - qemu-img map: Don't limit block status request size > iotest 114 fails on FreeBSD and OpenBSD (and probably NetBSD but that build hasn't reported back yet): looks like a non-portable use of 'truncate' ? TESTiotest-qcow2: 114 [fail] QEMU -- "/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64" -nodefaults -disp lay none -machine virt -accel qtest QEMU_IMG -- "/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-img" QEMU_IO -- "/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-io" --cache writeback --aio threads -f qcow2 QEMU_NBD -- "/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-nbd" IMGFMT-- qcow2 (compat=1.1) IMGPROTO -- file PLATFORM -- FreeBSD/amd64 freebsd 12.1-RELEASE TEST_DIR -- /home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch SOCK_DIR -- /tmp/tmp.RQQ2LyuG SOCKET_SCM_HELPER -- --- /home/qemu/qemu-test.FOdDM2/src/tests/qemu-iotests/114.out 2020-07-10 14:51:21.0 + +++ /home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/114.out.bad 2020-07-10 15:15:14.446278000 + @@ -1,8 +1,11 @@ QA output created by 114 -qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of raw) -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.orig +truncate: illegal option -- - +usage: truncate [-c] -s [+|-|%|/]size[K|k|M|m|G|g|T|t] file ... + truncate [-c] -r rfile file ... +qemu-img: TEST_DIR/t.IMGFMT: Could not open 'TEST_DIR/t.IMGFMT.orig': No such file or directory +Could not open backing image. Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 -qemu-img: warning: Deprecated use of backing file without explicit backing format +qemu-img: Could not open '/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch/t.qcow2.orig': Could not open '/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch/t.qcow2.orig': No such file or directory qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of IMGFMT) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT qemu-img: warning: Deprecated use of unopened backing file without explicit backing format, use of this image requires potentially unsafe format probing thanks -- PMM
Re: [PULL v2] Block layer patches
On 7/10/20 10:41 AM, Peter Maydell wrote: qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200) Block layer patches: - file-posix: Mitigate file fragmentation with extent size hints - Tighten qemu-img rules on missing backing format - qemu-img map: Don't limit block status request size iotest 114 fails on FreeBSD and OpenBSD (and probably NetBSD but that build hasn't reported back yet): looks like a non-portable use of 'truncate' ? @@ -1,8 +1,11 @@ QA output created by 114 -qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of raw) -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.orig +truncate: illegal option -- - +usage: truncate [-c] -s [+|-|%|/]size[K|k|M|m|G|g|T|t] file ... + truncate [-c] -r rfile file ... +qemu-img: TEST_DIR/t.IMGFMT: Could not open 'TEST_DIR/t.IMGFMT.orig': Yep, 'truncate' is not POSIX, but we've been lucky that most of our uses have been under '_supported_os Linux'; test 272 runs on BSD with 'truncate -s' instead of 'truncate --size'. The fix is obvious; I can post the followup to squash in if that will help. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PULL v2] Block layer patches
On 7/9/20 7:17 AM, Kevin Wolf wrote: The following changes since commit 8796c64ecdfd34be394ea277a53df0c76996: Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1: qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200) Block layer patches: - file-posix: Mitigate file fragmentation with extent size hints - Tighten qemu-img rules on missing backing format - qemu-img map: Don't limit block status request size Eric Blake (10): qemu-img: Flush stdout before before potential stderr messages block: Finish deprecation of 'qemu-img convert -n -o' sheepdog: Add trivial backing_fmt support vmdk: Add trivial backing_fmt support qcow: Tolerate backing_fmt= I see you renumbered my test 293 to 301 for conflict resolution, but did not update the commit message to call out the updated number ;) Since we may have to do a v3 because of the 'truncate -s' issue in 114, you could touch that up too. tests/qemu-iotests/301 | 88 tests/qemu-iotests/301.out | 59 +++ tests/qemu-iotests/common.filter | 62 tests/qemu-iotests/group | 1 + 146 files changed, 906 insertions(+), 472 deletions(-) create mode 100755 tests/qemu-iotests/301 create mode 100644 tests/qemu-iotests/301.out -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
