Re: [Qemu-block] [PATCH v3 6/7] iotests: Test qemu-img convert --salvage

[Max Reitz]

On 16.04.19 10:02, Vladimir Sementsov-Ogievskiy wrote:
> 13.04.2019 19:53, Max Reitz wrote:
>> This test converts a simple image to another, but blkdebug injects
>> block_status and read faults at some offsets.  The resulting image
>> should be the same as the input image, except that sectors that could
>> not be read have to be 0.
>>
>> Signed-off-by: Max Reitz 
>> ---
>>   tests/qemu-iotests/249 | 163 +
>>   tests/qemu-iotests/249.out |  43 ++
>>   tests/qemu-iotests/group   |   1 +
>>   3 files changed, 207 insertions(+)
>>   create mode 100755 tests/qemu-iotests/249
>>   create mode 100644 tests/qemu-iotests/249.out
>>
>> diff --git a/tests/qemu-iotests/249 b/tests/qemu-iotests/249
>> new file mode 100755
>> index 00..d3883d75f3
>> --- /dev/null
>> +++ b/tests/qemu-iotests/249
>> @@ -0,0 +1,163 @@
>> +#!/bin/bash
>> +#
>> +# Test qemu-img convert --salvage
>> +#
>> +# Copyright (C) 2018 Red Hat, Inc.
> 
> 2019
> 
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program.  If not, see .
>> +#
>> +
>> +# creator
>> +owner=mre...@redhat.com
>> +
>> +seq=$(basename $0)
>> +echo "QA output created by $seq"
>> +
>> +here=$PWD
> 
> this variable is not defined in latest bash-based iotests, is it needed 
> really?

Hm, no, it’s just here to remind me that this test really is from 2018. :-)

>> +status=1# failure is the default!
>> +
>> +_cleanup()
>> +{
>> +_cleanup_test_img
> 
> as I understand, you also need to remove source img which is $TEST_IMG.orig

That’s done automatically by _cleanup_test_img().

(It removes $TEST_IMG, $TEST_IMG.orig, and $TEST_IMG.base.)

>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# get standard environment, filters and checks
>> +. ./common.rc
>> +. ./common.filter
>> +. ./common.qemu
>> +
>> +_supported_fmt generic
>> +_supported_proto file
>> +_supported_os Linux
>> +
>> +
>> +TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
>> +
>> +$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
>> +
>> +
>> +sector_size=512
>> +
>> +# Offsets on which to fail block-status.  Keep in ascending order so
>> +# the indexing done by _filter_offsets will appear in ascending order
>> +# in the output as well.
>> +status_fail_offsets="$((16 * 1024 * 1024 + 8192))
>> + $((33 * 1024 * 1024 + 512))"
>> +
>> +# Offsets on which to fail reads.  Keep in ascending order for the
>> +# same reason.
>> +# Element 1 is shared with $status_fail_offsets on purpose.
> 
> you mean element 2
> 
>> +# Elements 2 and later test what happens when a continuous range of
> 
> and 3..
> 
> ah no, you just counting from zero..

Of course I am. :-)

I can rewrite the comment to "The second element" and "Starting with the
third element, we test...".

>> +# sectors is inaccessible.
>> +read_fail_offsets="$((32 * 1024 * 1024 - 65536))
>> +   $((33 * 1024 * 1024 + 512))
>> +   $(seq $((34 * 1024 * 1024)) $sector_size \
>> + $((34 * 1024 * 1024 + 4096 - $sector_size)))"
>> +
>> +
>> +# blkdebug must be above the format layer so it can intercept all
>> +# block-status events
>> +source_img="json:{'driver': 'blkdebug',
>> +  'image': {
>> +  'driver': '$IMGFMT',
>> +  'file': {
>> +  'driver': 'file',
>> +  'filename': '$TEST_IMG.orig'
>> +  }
>> +  },
>> +  'inject-error': ["
>> +
>> +for ofs in $status_fail_offsets
>> +do
>> +source_img+="{ 'event': 'none',
> 
> idea: may be make @event optional with default of 'none'?

I don’t know whether that makes sense.  'none' is a special case, so my
personal feeling is not to make it the default.

>> +   'iotype': 'block-status',
>> +   'errno': 5,
>> +   'sector': $((ofs / sector_size)) },"
>> +done
>> +
>> +for ofs in $read_fail_offsets
>> +do
>> +source_img+="{ 'event': 'none',
>> +   'iotype': 'read',
>> +   'errno': 5,
>> +   'sector': $((ofs / sector_size)) },"
>> +done
>> +
>> +# Remove the trailing comma and terminate @inject-error and json:{}
>> +source_img="${source_img%,} ] }"
>> +
>> +
>> +echo
>> +
>> +
>> +_filter_offsets() {
> 
> why to filter them? 

Re: [Qemu-block] [PATCH v3 6/7] iotests: Test qemu-img convert --salvage

[Vladimir Sementsov-Ogievskiy]

13.04.2019 19:53, Max Reitz wrote:
> This test converts a simple image to another, but blkdebug injects
> block_status and read faults at some offsets.  The resulting image
> should be the same as the input image, except that sectors that could
> not be read have to be 0.
> 
> Signed-off-by: Max Reitz 
> ---
>   tests/qemu-iotests/249 | 163 +
>   tests/qemu-iotests/249.out |  43 ++
>   tests/qemu-iotests/group   |   1 +
>   3 files changed, 207 insertions(+)
>   create mode 100755 tests/qemu-iotests/249
>   create mode 100644 tests/qemu-iotests/249.out
> 
> diff --git a/tests/qemu-iotests/249 b/tests/qemu-iotests/249
> new file mode 100755
> index 00..d3883d75f3
> --- /dev/null
> +++ b/tests/qemu-iotests/249
> @@ -0,0 +1,163 @@
> +#!/bin/bash
> +#
> +# Test qemu-img convert --salvage
> +#
> +# Copyright (C) 2018 Red Hat, Inc.

2019

> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see .
> +#
> +
> +# creator
> +owner=mre...@redhat.com
> +
> +seq=$(basename $0)
> +echo "QA output created by $seq"
> +
> +here=$PWD

this variable is not defined in latest bash-based iotests, is it needed really?


> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> +_cleanup_test_img

as I understand, you also need to remove source img which is $TEST_IMG.orig

> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.qemu
> +
> +_supported_fmt generic
> +_supported_proto file
> +_supported_os Linux
> +
> +
> +TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
> +
> +$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
> +
> +
> +sector_size=512
> +
> +# Offsets on which to fail block-status.  Keep in ascending order so
> +# the indexing done by _filter_offsets will appear in ascending order
> +# in the output as well.
> +status_fail_offsets="$((16 * 1024 * 1024 + 8192))
> + $((33 * 1024 * 1024 + 512))"
> +
> +# Offsets on which to fail reads.  Keep in ascending order for the
> +# same reason.
> +# Element 1 is shared with $status_fail_offsets on purpose.

you mean element 2

> +# Elements 2 and later test what happens when a continuous range of

and 3..

ah no, you just counting from zero..

> +# sectors is inaccessible.
> +read_fail_offsets="$((32 * 1024 * 1024 - 65536))
> +   $((33 * 1024 * 1024 + 512))
> +   $(seq $((34 * 1024 * 1024)) $sector_size \
> + $((34 * 1024 * 1024 + 4096 - $sector_size)))"
> +
> +
> +# blkdebug must be above the format layer so it can intercept all
> +# block-status events
> +source_img="json:{'driver': 'blkdebug',
> +  'image': {
> +  'driver': '$IMGFMT',
> +  'file': {
> +  'driver': 'file',
> +  'filename': '$TEST_IMG.orig'
> +  }
> +  },
> +  'inject-error': ["
> +
> +for ofs in $status_fail_offsets
> +do
> +source_img+="{ 'event': 'none',

idea: may be make @event optional with default of 'none'?

> +   'iotype': 'block-status',
> +   'errno': 5,
> +   'sector': $((ofs / sector_size)) },"
> +done
> +
> +for ofs in $read_fail_offsets
> +do
> +source_img+="{ 'event': 'none',
> +   'iotype': 'read',
> +   'errno': 5,
> +   'sector': $((ofs / sector_size)) },"
> +done
> +
> +# Remove the trailing comma and terminate @inject-error and json:{}
> +source_img="${source_img%,} ] }"
> +
> +
> +echo
> +
> +
> +_filter_offsets() {

why to filter them? They are constant..

> +filters=
> +
> +index=0
> +for ofs in $2
> +do
> +filters+=" -e s/$(printf "$1" $ofs)/status_fail_offset_$index/"
> +index=$((index + 1))
> +done
> +
> +index=0
> +for ofs in $3
> +do
> +filters+=" -e s/$(printf "$1" $ofs)/read_fail_offset_$index/"
> +index=$((index + 1))
> +done
> +
> +sed $filters
> +}
> +
> +# While determining the number of allocated sectors in the input
> +# image, we should see one block status warning per element of
> +# $status_fail_offsets.
> +#
> +# Then, the image is read.  Since the block status is queried in
> +# basic

[Qemu-block] [PATCH v3 6/7] iotests: Test qemu-img convert --salvage

[Max Reitz]

This test converts a simple image to another, but blkdebug injects
block_status and read faults at some offsets.  The resulting image
should be the same as the input image, except that sectors that could
not be read have to be 0.

Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/249 | 163 +
 tests/qemu-iotests/249.out |  43 ++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 207 insertions(+)
 create mode 100755 tests/qemu-iotests/249
 create mode 100644 tests/qemu-iotests/249.out

diff --git a/tests/qemu-iotests/249 b/tests/qemu-iotests/249
new file mode 100755
index 00..d3883d75f3
--- /dev/null
+++ b/tests/qemu-iotests/249
@@ -0,0 +1,163 @@
+#!/bin/bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=mre...@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+here=$PWD
+status=1   # failure is the default!
+
+_cleanup()
+{
+_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+ $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# Element 1 is shared with $status_fail_offsets on purpose.
+# Elements 2 and later test what happens when a continuous range of
+# sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+   $((33 * 1024 * 1024 + 512))
+   $(seq $((34 * 1024 * 1024)) $sector_size \
+ $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+  'image': {
+  'driver': '$IMGFMT',
+  'file': {
+  'driver': 'file',
+  'filename': '$TEST_IMG.orig'
+  }
+  },
+  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+source_img+="{ 'event': 'none',
+   'iotype': 'block-status',
+   'errno': 5,
+   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+source_img+="{ 'event': 'none',
+   'iotype': 'read',
+   'errno': 5,
+   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+
+echo
+
+
+_filter_offsets() {
+filters=
+
+index=0
+for ofs in $2
+do
+filters+=" -e s/$(printf "$1" $ofs)/status_fail_offset_$index/"
+index=$((index + 1))
+done
+
+index=0
+for ofs in $3
+do
+filters+=" -e s/$(printf "$1" $ofs)/read_fail_offset_$index/"
+index=$((index + 1))
+done
+
+sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+| _filter_offsets '%i' "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output ima