Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
On 10/09/2018 01:21 AM, Max Reitz wrote: > On 09.10.18 00:14, Vladimir Sementsov-Ogievskiy wrote: >> >> >> On 10/09/2018 01:08 AM, Max Reitz wrote: >>> On 09.10.18 00:02, Vladimir Sementsov-Ogievskiy wrote: On 10/08/2018 11:51 PM, Max Reitz wrote: > On 17.08.18 14:22, Vladimir

Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Max Reitz
On 09.10.18 00:14, Vladimir Sementsov-Ogievskiy wrote: > > > On 10/09/2018 01:08 AM, Max Reitz wrote: >> On 09.10.18 00:02, Vladimir Sementsov-Ogievskiy wrote: >>> >>> >>> On 10/08/2018 11:51 PM, Max Reitz wrote: On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Rewrite corrupted

Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
On 10/09/2018 01:08 AM, Max Reitz wrote: > On 09.10.18 00:02, Vladimir Sementsov-Ogievskiy wrote: >> >> >> On 10/08/2018 11:51 PM, Max Reitz wrote: >>> On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: Rewrite corrupted L2 table entry, which reference space out of underlying file.

Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Max Reitz
On 09.10.18 00:02, Vladimir Sementsov-Ogievskiy wrote: > > > On 10/08/2018 11:51 PM, Max Reitz wrote: >> On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: >>> Rewrite corrupted L2 table entry, which reference space out of >>> underlying file. >>> >>> Make this L2 table entry

Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
On 10/08/2018 11:51 PM, Max Reitz wrote: > On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: >> Rewrite corrupted L2 table entry, which reference space out of >> underlying file. >> >> Make this L2 table entry read-as-all-zeros without any allocation. >> >> Signed-off-by: Vladimir

Re: [Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Rewrite corrupted L2 table entry, which reference space out of > underlying file. > > Make this L2 table entry read-as-all-zeros without any allocation. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/qcow2-refcount.c | 32

Re: [Qemu-block] [PATCH v2 2/7] block/qcow2-refcount: avoid eating RAM

2018-10-08 Thread Max Reitz
On 08.10.18 22:22, Vladimir Sementsov-Ogievskiy wrote: > > > On 10/08/2018 06:31 PM, Max Reitz wrote: >> On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: >>> qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat >>> an unpredictable amount of memory on corrupted table

Re: [Qemu-block] [PATCH v2 2/7] block/qcow2-refcount: avoid eating RAM

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
On 10/08/2018 06:31 PM, Max Reitz wrote: > On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: >> qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat >> an unpredictable amount of memory on corrupted table entries, which are >> referencing regions far beyond the end of file.

Re: [Qemu-block] [PATCH v11 07/31] iotests.py: Add node_info()

2018-10-08 Thread Max Reitz
On 08.10.18 21:59, John Snow wrote: > > > On 10/08/2018 03:57 PM, Max Reitz wrote: >> On 08.10.18 21:34, John Snow wrote: >>> >>> >>> On 10/05/2018 07:39 PM, Max Reitz wrote: This function queries a node; since we cannot do that right now, it executes query-named-block-nodes and

Re: [Qemu-block] [PATCH v2 2/7] block/qcow2-refcount: avoid eating RAM

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
On 10/08/2018 06:31 PM, Max Reitz wrote: > On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: >> qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat >> an unpredictable amount of memory on corrupted table entries, which are >> referencing regions far beyond the end of file.

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/2] MAINTAINERS: Replace myself with John Snow for block jobs

2018-10-08 Thread John Snow
On 09/26/2018 02:05 PM, Jeff Cody wrote: > I'll not be involved with day-to-day qemu development, and John > Snow is a block jobs wizard. Have him take over block job > maintainership duties. > ... I didn't realize he wrote it like this, and I am told I should accept compliments, ... Well,

Re: [Qemu-block] [PATCH v2 6/7] block/qcow2-refcount: fix out-of-file L1 entries to be zero

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Zero out corrupted L1 table entry, which reference L2 table out of > underlying file. > Zero L1 table entry means that "the L2 table and all clusters described > by this L2 table are unallocated." > > Signed-off-by: Vladimir

Re: [Qemu-block] [PATCH v11 07/31] iotests.py: Add node_info()

2018-10-08 Thread John Snow
On 10/08/2018 03:57 PM, Max Reitz wrote: > On 08.10.18 21:34, John Snow wrote: >> >> >> On 10/05/2018 07:39 PM, Max Reitz wrote: >>> This function queries a node; since we cannot do that right now, it >>> executes query-named-block-nodes and returns the matching node's object. >>> >>>

Re: [Qemu-block] [PATCH v11 07/31] iotests.py: Add node_info()

2018-10-08 Thread Max Reitz
On 08.10.18 21:34, John Snow wrote: > > > On 10/05/2018 07:39 PM, Max Reitz wrote: >> This function queries a node; since we cannot do that right now, it >> executes query-named-block-nodes and returns the matching node's object. >> >> Signed-off-by: Max Reitz >> --- >>

Re: [Qemu-block] [PATCH v2 5/7] block/qcow2-refcount: check_refcounts_l2: split fix_l2_entry_to_zero

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Split entry repairing to separate function, to be reused later. > > Note: entry in in-memory l2 table (local variable in > check_refcounts_l2) is not updated after this patch. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- >

Re: [Qemu-block] [Qemu-devel] [PATCH] tests: Disable test-bdrv-drain

2018-10-08 Thread Eric Blake
On 10/8/18 11:40 AM, Kevin Wolf wrote: Am 08.10.2018 um 17:43 hat Peter Maydell geschrieben: Looking at the backtraces I'm wondering if this is the result of an implicit reliance on the order in which per-thread destructors are called (which is left unspecified by POSIX) -- the destructor

Re: [Qemu-block] [PATCH 05/10] docs/devel/testing.rst: add missing newlines after code block

2018-10-08 Thread John Snow
On 10/04/2018 12:18 PM, Cleber Rosa wrote: > The line immediate following a ".. code::" block is considered > to contains arguments to the "code directive". The lack of a > new line gives me during at parse time: > >testing.rst:63: (ERROR/3) Error in "code" directive: >maximum 1

Re: [Qemu-block] [PATCH 09/10] scripts/qemu.py: use a more consistent docstring style

2018-10-08 Thread John Snow
On 10/04/2018 12:18 PM, Cleber Rosa wrote: > Signed-off-by: Cleber Rosa > --- > dtc | 2 +- > scripts/qemu.py | 65 +++-- > 2 files changed, 42 insertions(+), 25 deletions(-) > > diff --git a/dtc b/dtc > index 88f18909db..e54388015a

Re: [Qemu-block] [PATCH v11 07/31] iotests.py: Add node_info()

2018-10-08 Thread John Snow
On 10/05/2018 07:39 PM, Max Reitz wrote: > This function queries a node; since we cannot do that right now, it > executes query-named-block-nodes and returns the matching node's object. > > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/iotests.py | 7 +++ > 1 file changed, 7

Re: [Qemu-block] [PATCH 14/14] block: Stop passing flags to bdrv_reopen_queue_child()

2018-10-08 Thread Max Reitz
On 08.10.18 20:13, Alberto Garcia wrote: > On Mon 08 Oct 2018 04:48:50 AM CEST, Max Reitz wrote: > >>> +/* Old values are used for options that aren't set yet */ >>> +old_options = qdict_clone_shallow(bs->options); >>> +bdrv_join_options(bs, options, old_options); >>> +

Re: [Qemu-block] [PATCH 14/14] block: Stop passing flags to bdrv_reopen_queue_child()

2018-10-08 Thread Alberto Garcia
On Mon 08 Oct 2018 04:48:50 AM CEST, Max Reitz wrote: >> +/* Old values are used for options that aren't set yet */ >> +old_options = qdict_clone_shallow(bs->options); >> +bdrv_join_options(bs, options, old_options); >> +qobject_unref(old_options); >> + >> +/* We have the

Re: [Qemu-block] [PATCH 08/14] block: Use bdrv_reopen_set_read_only() in external_snapshot_commit()

2018-10-08 Thread Alberto Garcia
On Mon 08 Oct 2018 03:31:08 AM CEST, Max Reitz wrote: > On 19.09.18 16:47, Alberto Garcia wrote: >> This patch replaces the bdrv_reopen() calls that set and remove the >> BDRV_O_RDWR flag with the new bdrv_reopen_set_read_only() function. > > Ha! Got you! It's just one call this time, not

Re: [Qemu-block] [Qemu-devel] [PATCH] tests: Disable test-bdrv-drain

2018-10-08 Thread Peter Maydell
On 8 October 2018 at 17:40, Kevin Wolf wrote: > By the way, can you reproduce this with virtio-blk/scsi and an iothread > in a real QEMU or is it only the test case that fails? In theory, I > don't see what would prevent QEMU from hanging at shutdown. I haven't tested, but I suspect this is less

Re: [Qemu-block] [PATCH v2 2/2] MAINTAINERS: Remove myself as block maintainer

2018-10-08 Thread John Snow
On 10/07/2018 07:51 PM, Max Reitz wrote: > On 26.09.18 20:05, Jeff Cody wrote: >> I'll not be involved in day-to-day qemu development. Remove myself as >> maintainer from the remainder of the network block drivers, and revert >> them to the general block layer maintainership. >> >> Move

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Kevin Wolf
Am 08.10.2018 um 18:04 hat Anton Nefedov geschrieben: > > > On 8/10/2018 6:46 PM, Kevin Wolf wrote: > > Am 08.10.2018 um 17:25 hat Anton Nefedov geschrieben: > >> > >> > >> On 8/10/2018 6:03 PM, Kevin Wolf wrote: > >>> Am 08.10.2018 um 16:38 hat Anton Nefedov geschrieben: > On 4/10/2018

Re: [Qemu-block] [Qemu-devel] [PATCH] tests: Disable test-bdrv-drain

2018-10-08 Thread Kevin Wolf
Am 08.10.2018 um 17:43 hat Peter Maydell geschrieben: > Looking at the backtraces I'm wondering if this is the result of > an implicit reliance on the order in which per-thread destructors > are called (which is left unspecified by POSIX) -- the destructor > function qemu_thread_atexit_run() is

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Anton Nefedov
On 8/10/2018 6:46 PM, Kevin Wolf wrote: > Am 08.10.2018 um 17:25 hat Anton Nefedov geschrieben: >> >> >> On 8/10/2018 6:03 PM, Kevin Wolf wrote: >>> Am 08.10.2018 um 16:38 hat Anton Nefedov geschrieben: On 4/10/2018 6:33 PM, Kevin Wolf wrote: > Am 21.08.2018 um 11:46 hat Anton Nefedov

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Kevin Wolf
Am 08.10.2018 um 17:25 hat Anton Nefedov geschrieben: > > > On 8/10/2018 6:03 PM, Kevin Wolf wrote: > > Am 08.10.2018 um 16:38 hat Anton Nefedov geschrieben: > >> On 4/10/2018 6:33 PM, Kevin Wolf wrote: > >>> Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: > Signed-off-by: Anton

Re: [Qemu-block] [PATCH v2 4/7] block/qcow2-refcount: check_refcounts_l2: reduce ignored overlaps

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Reduce number of structures ignored in overlap check: when checking > active table ignore active tables, when checking inactive table ignore > inactive ones. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/qcow2-refcount.c |

Re: [Qemu-block] [Qemu-devel] [PATCH] tests: Disable test-bdrv-drain

2018-10-08 Thread Peter Maydell
On 8 October 2018 at 10:12, Peter Maydell wrote: > I looked back at the backtrace/etc that I posted earlier in this > thread, and it looked to me like maybe a memory corruption issue. > So I tried running the test under valgrind on Linux, and: ...which goes away if I do a complete build from

Re: [Qemu-block] [PATCH v2 3/7] block/qcow2-refcount: check_refcounts_l2: refactor compressed case

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Separate offset and size of compressed cluster. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/qcow2-refcount.c | 15 ++- > 1 file changed, 10 insertions(+), 5 deletions(-) > > diff --git

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Anton Nefedov
On 8/10/2018 6:03 PM, Kevin Wolf wrote: > Am 08.10.2018 um 16:38 hat Anton Nefedov geschrieben: >> On 4/10/2018 6:33 PM, Kevin Wolf wrote: >>> Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: Signed-off-by: Anton Nefedov Reviewed-by: Alberto Garcia --- hw/ide/core.c

Re: [Qemu-block] [PATCH v2 2/7] block/qcow2-refcount: avoid eating RAM

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat > an unpredictable amount of memory on corrupted table entries, which are > referencing regions far beyond the end of file. > > Prevent this, by skipping such regions

Re: [Qemu-block] [PATCH v2 1/7] block/qcow2-refcount: fix check_oflag_copied

2018-10-08 Thread Max Reitz
On 17.08.18 14:22, Vladimir Sementsov-Ogievskiy wrote: > Increase corruptions_fixed only after successful fix. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/qcow2-refcount.c | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/block/qcow2-refcount.c

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Kevin Wolf
Am 08.10.2018 um 16:38 hat Anton Nefedov geschrieben: > On 4/10/2018 6:33 PM, Kevin Wolf wrote: > > Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: > >> Signed-off-by: Anton Nefedov > >> Reviewed-by: Alberto Garcia > >> --- > >> hw/ide/core.c | 12 > >> 1 file changed, 12

Re: [Qemu-block] [PATCH 0/7] qcow2 check improvements

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
ping Hi, what about this? 17.08.2018 15:22, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > v2: > 02, 06: check bdrv_getlength error return code > > v1: > > We've faced the following problem: after host fs corruption, vm images > becomes invalid. And which is interesting, starting qemu-img

Re: [Qemu-block] [PATCH v4 3/8] ide: account UNMAP (TRIM) operations

2018-10-08 Thread Anton Nefedov
On 4/10/2018 6:33 PM, Kevin Wolf wrote: > Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: >> Signed-off-by: Anton Nefedov >> Reviewed-by: Alberto Garcia >> --- >> hw/ide/core.c | 12 >> 1 file changed, 12 insertions(+) >> >> diff --git a/hw/ide/core.c b/hw/ide/core.c >>

Re: [Qemu-block] [PATCH v4 6/8] scsi: account unmap operations

2018-10-08 Thread Anton Nefedov
On 4/10/2018 6:47 PM, Kevin Wolf wrote: > Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: >> Signed-off-by: Anton Nefedov >> Reviewed-by: Vladimir Sementsov-Ogievskiy >> Reviewed-by: Alberto Garcia >> --- >> hw/scsi/scsi-disk.c | 9 - >> 1 file changed, 8 insertions(+), 1

Re: [Qemu-block] [PATCH v4 7/8] file-posix: account discard operations

2018-10-08 Thread Anton Nefedov
On 4/10/2018 6:52 PM, Kevin Wolf wrote: > Am 21.08.2018 um 11:46 hat Anton Nefedov geschrieben: >> This will help to identify how many of the user-issued discard operations >> (accounted on a device level) have actually suceeded down on the host file >> (even though the numbers will not be

Re: [Qemu-block] [PATCH v2 5/6] qapi: add block-dirty-bitmap-merge

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
20.09.2018 22:40, Eric Blake wrote: > [reviving an old patch] > > On 1/16/18 6:54 AM, Vladimir Sementsov-Ogievskiy wrote: >> Signed-off-by: Vladimir Sementsov-Ogievskiy >> --- >>   qapi/block-core.json | 38 >> ++ >>   include/block/dirty-bitmap.h |  2

Re: [Qemu-block] [PATCH v3 1/3] qapi: add x-debug-query-block-graph

2018-10-08 Thread Vladimir Sementsov-Ogievskiy
05.10.2018 22:34, Max Reitz wrote: On 02.10.18 15:01, Vladimir Sementsov-Ogievskiy wrote: 28.09.2018 19:31, Max Reitz wrote: On 23.08.18 17:46, Vladimir Sementsov-Ogievskiy wrote: Add a new command, returning block nodes (and their users) graph. Signed-off-by: Vladimir Sementsov-Ogievskiy

Re: [Qemu-block] [Qemu-devel] [PATCH] tests: Disable test-bdrv-drain

2018-10-08 Thread Peter Maydell
On 5 October 2018 at 19:09, Kevin Wolf wrote: > And if we disable it wholesale, then nobody has any incentive to fix any > bug that the test case could have uncovered. Yes, that's fair. I'm sorry; I was a bit grumpy when I wrote that email because my test runs had been bumping into it all day.

Re: [Qemu-block] [PATCH] qemu-io-cmds: Fix two format strings

2018-10-08 Thread Kevin Wolf
Am 06.10.2018 um 20:38 hat Stefan Weil geschrieben: > Use %zu instead of %zd for unsigned numbers. > > This fixes two error messages from the LSTM static code analyzer: > > This argument should be of type 'ssize_t' but is of type 'unsigned long' > > Signed-off-by: Stefan Weil Thanks, applied