Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Eric Blake writes:
> On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
>> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
>> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
>> > report over-allocation to du (based on the hueristics the filesystem
>> > is making while observing the contents being mirrored); even though
>> > the contents and actual size matched, about 50% of the time the size
>> > reported by disk_usage was too large by 4k, failing the test. In
>> > auditing other iotests, this is a common problem we've had to deal
>> > with.
>> >
>> > Reported-by: Fiona Ebner
>> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
>> > Signed-off-by: Eric Blake
>> > Reviewed-by: Fiona Ebner
>> > Tested-by: Fiona Ebner
>> >
>> > ---
>> > v2: Drop sync for ZFS
>> > ---
>> > tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>> > 1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> Squashing this in since Markus is reporting failures even with this
>> patch applied, and I'm lacking enough context to see what else might
>> be interfering:
>>
>> diff --git i/tests/qemu-iotests/tests/mirror-sparse
>> w/tests/qemu-iotests/tests/mirror-sparse
>> index 39e3196c811..a30e52de41d 100755
>> --- i/tests/qemu-iotests/tests/mirror-sparse
>> +++ w/tests/qemu-iotests/tests/mirror-sparse
>> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
>> elif test $result -gt $((19*1024*1024)); then
>> actual=full
>> else
>> -actual=unknown
>> +actual="unexpected size ($result)"
>
> I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
> install on Fedora) - it turns out that XFS rounds disk usage in terms
> of allocation groups which default to a size of 1M; so I need to treat
> exactly 3M as sparse, not unknown. I'm also squashing in:
>
> diff --git i/tests/qemu-iotests/tests/mirror-sparse
> w/tests/qemu-iotests/tests/mirror-sparse
> index a30e52de41d..cfcaa600ab4 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base
> $TEST_IMG
> # Some filesystems can fudge allocations for various reasons; rather
> # than expecting precise 2M and 20M images, it is better to allow for slop.
> result=$(disk_usage $TEST_IMG)
> -if test $result -lt $((3*1024*1024)); then
> +if test $result -lt $((4*1024*1024)); then
> actual=sparse
> elif test $result -gt $((19*1024*1024)); then
> actual=full
This one-liner appears to fix it for me. Thanks!
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
> > report over-allocation to du (based on the hueristics the filesystem
> > is making while observing the contents being mirrored); even though
> > the contents and actual size matched, about 50% of the time the size
> > reported by disk_usage was too large by 4k, failing the test. In
> > auditing other iotests, this is a common problem we've had to deal
> > with.
> >
> > Reported-by: Fiona Ebner
> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> > Signed-off-by: Eric Blake
> > Reviewed-by: Fiona Ebner
> > Tested-by: Fiona Ebner
> >
> > ---
> > v2: Drop sync for ZFS
> > ---
> > tests/qemu-iotests/tests/mirror-sparse | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
>
> Squashing this in since Markus is reporting failures even with this
> patch applied, and I'm lacking enough context to see what else might
> be interfering:
>
> diff --git i/tests/qemu-iotests/tests/mirror-sparse
> w/tests/qemu-iotests/tests/mirror-sparse
> index 39e3196c811..a30e52de41d 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
> elif test $result -gt $((19*1024*1024)); then
> actual=full
> else
> -actual=unknown
> +actual="unexpected size ($result)"
I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
install on Fedora) - it turns out that XFS rounds disk usage in terms
of allocation groups which default to a size of 1M; so I need to treat
exactly 3M as sparse, not unknown. I'm also squashing in:
diff --git i/tests/qemu-iotests/tests/mirror-sparse
w/tests/qemu-iotests/tests/mirror-sparse
index a30e52de41d..cfcaa600ab4 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base
$TEST_IMG
# Some filesystems can fudge allocations for various reasons; rather
# than expecting precise 2M and 20M images, it is better to allow for slop.
result=$(disk_usage $TEST_IMG)
-if test $result -lt $((3*1024*1024)); then
+if test $result -lt $((4*1024*1024)); then
actual=sparse
elif test $result -gt $((19*1024*1024)); then
actual=full
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem
> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test. In
> auditing other iotests, this is a common problem we've had to deal
> with.
>
> Reported-by: Fiona Ebner
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake
> Reviewed-by: Fiona Ebner
> Tested-by: Fiona Ebner
>
> ---
> v2: Drop sync for ZFS
> ---
> tests/qemu-iotests/tests/mirror-sparse | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
Squashing this in since Markus is reporting failures even with this
patch applied, and I'm lacking enough context to see what else might
be interfering:
diff --git i/tests/qemu-iotests/tests/mirror-sparse
w/tests/qemu-iotests/tests/mirror-sparse
index 39e3196c811..a30e52de41d 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
elif test $result -gt $((19*1024*1024)); then
actual=full
else
-actual=unknown
+actual="unexpected size ($result)"
fi
echo "Destination is $actual; expected $expected"
}
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem
and despite my other edits, I still missed fixing my typo on heuristics
> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test. In
> auditing other iotests, this is a common problem we've had to deal
> with.
>
> Reported-by: Fiona Ebner
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake
> Reviewed-by: Fiona Ebner
> Tested-by: Fiona Ebner
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
