Re: noholes and incremental send streamsize

[Henk Slager]

I have filed:

https://bugzilla.kernel.org/show_bug.cgi?id=121321

On Sat, Jun 25, 2016 at 1:17 AM, Henk Slager  wrote:
> Hi,
>
> the virtual machine images files I create and use are mostly sparse,
> so that not too much space on a filesystem with snapshots and on
> filesystems that are receive targets is used.
> But I noticed that with just starting up and shutting down a virtual
> machine, the difference between 2 nightly snapshots as generated by
> send -p is huge. It turns out that the cause is the noholes flag of
> the filesystem. I have set this flag to have a bit less metadata
> hopefully, but it turns out to have a very negative side effect.

As a workaround, I flipped the no-holes flag back to zero, so that for
new files the problem is avoided I assume.

btrfs-check detects this as a problem as extents with no 'metadata
administration' are found and no-holes flag is not set. But so far,
the kernel can handle the situation apparently, so I currently won't
re-create the multi-TB filesystems just for this issue.

> The sparse part of a file that is changed between 2 snapshots is added
> to the difference between the 2 snapshots when doing send -p, in full
> 'no-sparse' size (as zeros I think, as the data compresses extremely
> well).
> So in case of a 50G VM that has roughly 8G of actual written
> filesystem data (a new OS install), I saw 40G of generated btrfs
> sendstream data by just a start/stop of the VM.
>
> I think this behavior is not correct: Sparse is read as zeros normally
> AFAIK, so if the file locations that are sparse are the same for a
> snapshotted file and its parent, there should not be a difference.
>
> The following sequence shows the problem; if the btrfstune command is
> skipped, streamsize is as expected.
>
>
> uname -r
> btrfs --version
> truncate -s 5G /holetest.img
> losetup /dev/loop0 /holetest.img
> mkfs.btrfs /dev/loop0
>
> btrfstune -n /dev/loop0
>
> btrfs-show-super /dev/loop0 | grep incompat_flags
> mount /dev/loop0 /mnt
> btrfs sub create /mnt/vol1
> truncate -s 4G /mnt/vol1/test1
> btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro0
> dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
> sync
> btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro1
> dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
> sync
> btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro2
> btrfs send -p /mnt/vol1.ro1 /mnt/vol1.ro2 > /sendsize.btrfs
> ls -al /sendsize.btrfs
> umount /mnt
> losetup -d /dev/loop0
> rm /holetest.img /sendsize.btrfs
>
>
> => output
>
> # uname -r
> 4.7.0-rc4-kade
> # btrfs --version
> btrfs-progs v4.6.1
> # truncate -s 5G /holetest.img
> # losetup /dev/loop0 /holetest.img
> # mkfs.btrfs /dev/loop0
> btrfs-progs v4.5.3+20160516

I used an alias for 'btrfs', to point to the binary in the btrfs-progs
git, but forgot to alias mkfs.btrfs, but anyhow the problem is there
for multiple kernel/progs versions.

> See http://btrfs.wiki.kernel.org for more information.
>
> Performing full device TRIM (5.00GiB) ...
> Label:  (null)
> UUID:   338e948c-ee8c-4b9f-926e-b6f8fe140ae6
> Node size:  16384
> Sector size:4096
> Filesystem size:5.00GiB
> Block group profiles:
>  Data: single8.00MiB
>  Metadata: DUP 264.00MiB
>  System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>   IDSIZE  PATH
>1 5.00GiB  /dev/loop0
>
> #
> # btrfstune -n /dev/loop0
> #
> # btrfs-show-super /dev/loop0 | grep incompat_flags
> incompat_flags  0x341
> # mount /dev/loop0 /mnt
> # btrfs sub create /mnt/vol1
> Create subvolume '/mnt/vol1'
> # truncate -s 4G /mnt/vol1/test1
> # btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro0
> Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro0'
> # dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1,0 MB, 1,0 MiB) copied, 0,0779673 s, 13,4 MB/s
> # sync
> # btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro1
> Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro1'
> # dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1,0 MB, 1,0 MiB) copied, 0,0634306 s, 16,5 MB/s
> # sync
> # btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro2
> Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro2'
> # btrfs send -p /mnt/vol1.ro1 /mnt/vol1.ro2 > /sendsize.btrfs
> At subvol /mnt/vol1.ro2
> # ls -al /sendsize.btrfs
> -rw-r--r-- 1 root root 4298025877 jun 25 00:26 /sendsize.btrfs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

noholes and incremental send streamsize

[Henk Slager]

Hi,

the virtual machine images files I create and use are mostly sparse,
so that not too much space on a filesystem with snapshots and on
filesystems that are receive targets is used.
But I noticed that with just starting up and shutting down a virtual
machine, the difference between 2 nightly snapshots as generated by
send -p is huge. It turns out that the cause is the noholes flag of
the filesystem. I have set this flag to have a bit less metadata
hopefully, but it turns out to have a very negative side effect.

The sparse part of a file that is changed between 2 snapshots is added
to the difference between the 2 snapshots when doing send -p, in full
'no-sparse' size (as zeros I think, as the data compresses extremely
well).
So in case of a 50G VM that has roughly 8G of actual written
filesystem data (a new OS install), I saw 40G of generated btrfs
sendstream data by just a start/stop of the VM.

I think this behavior is not correct: Sparse is read as zeros normally
AFAIK, so if the file locations that are sparse are the same for a
snapshotted file and its parent, there should not be a difference.

The following sequence shows the problem; if the btrfstune command is
skipped, streamsize is as expected.


uname -r
btrfs --version
truncate -s 5G /holetest.img
losetup /dev/loop0 /holetest.img
mkfs.btrfs /dev/loop0

btrfstune -n /dev/loop0

btrfs-show-super /dev/loop0 | grep incompat_flags
mount /dev/loop0 /mnt
btrfs sub create /mnt/vol1
truncate -s 4G /mnt/vol1/test1
btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro0
dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
sync
btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro1
dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
sync
btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro2
btrfs send -p /mnt/vol1.ro1 /mnt/vol1.ro2 > /sendsize.btrfs
ls -al /sendsize.btrfs
umount /mnt
losetup -d /dev/loop0
rm /holetest.img /sendsize.btrfs


=> output

# uname -r
4.7.0-rc4-kade
# btrfs --version
btrfs-progs v4.6.1
# truncate -s 5G /holetest.img
# losetup /dev/loop0 /holetest.img
# mkfs.btrfs /dev/loop0
btrfs-progs v4.5.3+20160516
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (5.00GiB) ...
Label:  (null)
UUID:   338e948c-ee8c-4b9f-926e-b6f8fe140ae6
Node size:  16384
Sector size:4096
Filesystem size:5.00GiB
Block group profiles:
 Data: single8.00MiB
 Metadata: DUP 264.00MiB
 System:   DUP  12.00MiB
SSD detected:   no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
  IDSIZE  PATH
   1 5.00GiB  /dev/loop0

#
# btrfstune -n /dev/loop0
#
# btrfs-show-super /dev/loop0 | grep incompat_flags
incompat_flags  0x341
# mount /dev/loop0 /mnt
# btrfs sub create /mnt/vol1
Create subvolume '/mnt/vol1'
# truncate -s 4G /mnt/vol1/test1
# btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro0
Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro0'
# dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
1+0 records in
1+0 records out
1048576 bytes (1,0 MB, 1,0 MiB) copied, 0,0779673 s, 13,4 MB/s
# sync
# btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro1
Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro1'
# dd if=/dev/urandom of=/mnt/vol1/test1 bs=1M count=1 conv=notrunc
1+0 records in
1+0 records out
1048576 bytes (1,0 MB, 1,0 MiB) copied, 0,0634306 s, 16,5 MB/s
# sync
# btrfs sub snap -r /mnt/vol1 /mnt/vol1.ro2
Create a readonly snapshot of '/mnt/vol1' in '/mnt/vol1.ro2'
# btrfs send -p /mnt/vol1.ro1 /mnt/vol1.ro2 > /sendsize.btrfs
At subvol /mnt/vol1.ro2
# ls -al /sendsize.btrfs
-rw-r--r-- 1 root root 4298025877 jun 25 00:26 /sendsize.btrfs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html