Re: [Qemu-devel] [PATCH v6 5/6] iotests: Add new test 214 for max compressed cluster offset

[Alberto Garcia]

On Tue 19 Jun 2018 08:51:36 PM CEST, Eric Blake wrote:
> On 04/26/2018 07:10 AM, Alberto Garcia wrote:
>> On Thu 26 Apr 2018 04:51:28 AM CEST, Eric Blake wrote:
>>> If you have a capable file system (tmpfs is good, ext4 not so much;
>>> run ./check with TEST_DIR pointing to a good location so as not
>>> to skip the test), it's actually possible to create a qcow2 file
>>> that expands to a sparse 512T image with just over 38M of content.
>>> The test is not the world's fastest (qemu crawling through 256M
>>> bits of refcount table to find the next cluster to allocate takes
>>> several seconds, as does qemu-img check reporting millions of
>>> leaked clusters); but it DOES catch the problem that the previous
>>> patch just fixed where writing a compressed cluster to a full
>>> image ended up overwriting the wrong cluster.
>>>
>>> Suggested-by: Max Reitz 
>>> Signed-off-by: Eric Blake 
>> 
>> Nice test :-)
>> 
>> Reviewed-by: Alberto Garcia 
>
> 214 is already in the tree in the meantime; this will need a rebase to
> pick the next available test number (220 might be claimed, so 222?)

You can also take 220 if you want, I suspect that it will take a bit
longer to merge the blockdev-reopen patches. Whatever is easier for you.

Berto

Re: [Qemu-devel] [PATCH v6 5/6] iotests: Add new test 214 for max compressed cluster offset

[Eric Blake]

On 04/26/2018 07:10 AM, Alberto Garcia wrote:

On Thu 26 Apr 2018 04:51:28 AM CEST, Eric Blake wrote:

If you have a capable file system (tmpfs is good, ext4 not so much;
run ./check with TEST_DIR pointing to a good location so as not
to skip the test), it's actually possible to create a qcow2 file
that expands to a sparse 512T image with just over 38M of content.
The test is not the world's fastest (qemu crawling through 256M
bits of refcount table to find the next cluster to allocate takes
several seconds, as does qemu-img check reporting millions of
leaked clusters); but it DOES catch the problem that the previous
patch just fixed where writing a compressed cluster to a full
image ended up overwriting the wrong cluster.

Suggested-by: Max Reitz 
Signed-off-by: Eric Blake 


Nice test :-)

Reviewed-by: Alberto Garcia 


214 is already in the tree in the meantime; this will need a rebase to 
pick the next available test number (220 might be claimed, so 222?)


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v6 5/6] iotests: Add new test 214 for max compressed cluster offset

[Alberto Garcia]

On Thu 26 Apr 2018 04:51:28 AM CEST, Eric Blake wrote:
> If you have a capable file system (tmpfs is good, ext4 not so much;
> run ./check with TEST_DIR pointing to a good location so as not
> to skip the test), it's actually possible to create a qcow2 file
> that expands to a sparse 512T image with just over 38M of content.
> The test is not the world's fastest (qemu crawling through 256M
> bits of refcount table to find the next cluster to allocate takes
> several seconds, as does qemu-img check reporting millions of
> leaked clusters); but it DOES catch the problem that the previous
> patch just fixed where writing a compressed cluster to a full
> image ended up overwriting the wrong cluster.
>
> Suggested-by: Max Reitz 
> Signed-off-by: Eric Blake 

Nice test :-)

Reviewed-by: Alberto Garcia 

Berto

[Qemu-devel] [PATCH v6 5/6] iotests: Add new test 214 for max compressed cluster offset

[Eric Blake]

If you have a capable file system (tmpfs is good, ext4 not so much;
run ./check with TEST_DIR pointing to a good location so as not
to skip the test), it's actually possible to create a qcow2 file
that expands to a sparse 512T image with just over 38M of content.
The test is not the world's fastest (qemu crawling through 256M
bits of refcount table to find the next cluster to allocate takes
several seconds, as does qemu-img check reporting millions of
leaked clusters); but it DOES catch the problem that the previous
patch just fixed where writing a compressed cluster to a full
image ended up overwriting the wrong cluster.

Suggested-by: Max Reitz 
Signed-off-by: Eric Blake 

---
v6: new patch; took over 90 seconds to run on my setup, using tmpfs
---
 tests/qemu-iotests/214 | 97 ++
 tests/qemu-iotests/214.out | 54 ++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 152 insertions(+)
 create mode 100755 tests/qemu-iotests/214
 create mode 100644 tests/qemu-iotests/214.out

diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214
new file mode 100755
index 000..b4127facd12
--- /dev/null
+++ b/tests/qemu-iotests/214
@@ -0,0 +1,97 @@
+#!/bin/bash
+#
+# max limits on compression in huge qcow2 files
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1# failure is the default!
+
+_cleanup()
+{
+_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.pattern
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+echo "== Creating huge file =="
+
+# Sanity check: We require a file system that permits the creation
+# of a HUGE (but very sparse) file.  tmpfs works, ext4 does not.
+if ! truncate --size=513T "$TEST_IMG"; then
+_notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+IMGOPTS='cluster_size=2M,refcount_bits=1' _make_test_img 513T
+
+echo "== Populating refcounts =="
+# We want an image with 256M refcounts * 2M clusters = 512T referenced.
+# Each 2M cluster holds 16M refcounts; the refcount table initially uses
+# 1 refblock, so we need to add 15 more.  The refcount table lives at 2M,
+# first refblock at 4M, L2 at 6M, so our remaining additions start at 8M.
+# Then, for each refblock, mark it as fully populated.
+to_hex() {
+printf %016x\\n $1 | sed 's/\(..\)/\\x\1/g'
+}
+truncate --size=38m "$TEST_IMG"
+entry=$((0x20))
+$QEMU_IO_PROG -f raw -c "w -P 0xff 4m 2m" "$TEST_IMG" | _filter_qemu_io
+for i in {1..15}; do
+offs=$((0x60 + i*0x20))
+poke_file "$TEST_IMG" $((i*8 + entry)) $(to_hex $offs)
+$QEMU_IO_PROG -f raw -c "w -P 0xff $offs 2m" "$TEST_IMG" | _filter_qemu_io
+done
+
+echo "== Checking file before =="
+# FIXME: 'qemu-img check' doesn't diagnose refcounts beyond the end of
+# the file as leaked clusters
+_check_test_img 2>&1 | sed '/^Leaked cluster/d'
+stat -c 'image size %s' "$TEST_IMG"
+
+echo "== Trying to write compressed cluster =="
+# Given our file size, the next available cluster at 512T lies beyond the
+# maximum offset that a compressed 2M cluster can reside in
+$QEMU_IO_PROG -c 'w -c 0 2m' "$TEST_IMG" | _filter_qemu_io
+# The attempt failed, but ended up allocating a new refblock
+stat -c 'image size %s' "$TEST_IMG"
+
+echo "== Writing normal cluster =="
+# The failed write should not corrupt the image, so a normal write succeeds
+$QEMU_IO_PROG -c 'w 0 2m' "$TEST_IMG" | _filter_qemu_io
+
+echo "== Checking file after =="
+# qemu-img now sees the millions of leaked clusters, thanks to the allocations
+# at 512T.  Undo many of our faked references to speed up the check.
+$QEMU_IO_PROG -f raw -c "w -z 5m 1m" -c "w -z 8m 30m" "$TEST_IMG" |
+_filter_qemu_io
+_check_test_img 2>&1 | sed '/^Leaked cluster/d'
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/214.out b/tests/qemu-iotests/214.out
new file mode 100644
index 000..38aafe1b961
--- /dev/null
+++ b/tests/qemu-iotests/214.out
@@ -0,0 +1,54 @@
+QA output created by 214
+== Creating huge file ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=564049465049088
+==