[linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
Hi, we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost. As a special condition, we are using the sector-size option with cryptsetup (LUKS2) to use 4096 byte sectors to increase the crypt performance. According to Ondrej Kozina in https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html this is due to the fact that a sector size of 4096 also causes the physical block size to be (at least) 4096. Using the default sector size of 512 does not show the problem. The source PV of the pvmove has a physical block size of 512 (SCSI disk in our case). So when moving this to the encrypted volume the physical block size becomes 4096. The file system does not seem to like this Please note that this problem can also happen in other cases, such as mixing disks with different block sizes (e.g. SCSI disks with 512 bytes and s390x-DASDs with 4096 block size). In my opinion this is a general problem that needs to be addressed by LVM. Whenever you pvmove data to a PV with a larger physical block size you will corrupt the file system and thus cause data loss. The same probably happens when using lvextend to enlarge an LV onto a PV with a larger physical block size. Bottom line: LVM should reject extending a volume group with a PV that has a larger physical block size than the PVs that belong to the volume group already. There might be a force option to override this check in case the user really knows what he does. But the default behavior should be to protect the user from file system corruptions and thus from data loss. Yes, having a backup before doing pvmove or similar is of course wise, but I still think that LVM should prevent a user for damaging the file system by using LVM's command line tools. Here is how to reproduce this (note the error messages on the very last command): # sudo dd if=/dev/zero of=loopbackfile1.img bs=500M count=1 1+0 records in 1+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 2.32777 s, 225 MB/s # sudo dd if=/dev/zero of=loopbackfile2.img bs=500M count=1 1+0 records in 1+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 1.89992 s, 276 MB/s # losetup -fP loopbackfile1.img # losetup -fP loopbackfile2.img # pvcreate /dev/loop0 Physical volume "/dev/loop0" successfully created. # vgcreate LOOP_VG /dev/loop0 Volume group "LOOP_VG" successfully created # lvcreate -L 300MB LOOP_VG -n LV /dev/loop0 Logical volume "LV" created. # mkfs.ext4 /dev/mapper/LOOP_VG-LV mke2fs 1.44.1 (24-Mar-2018) Discarding device blocks: done Creating filesystem with 307200 1k blocks and 76912 inodes Filesystem UUID: 344289a3-e251-4d88-b03d-a71a4be2a8ec Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729, 204801, 221185 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/mapper/LOOP_VG-LV /mnt # cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1 WARNING! This will overwrite data on /dev/loop1 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase for /dev/loop1: loop Verify passphrase: loop # cryptsetup luksOpen /dev/loop1 enc-loop Enter passphrase for /dev/loop1: loop # pvcreate /dev/mapper/enc-loop Physical volume "/dev/mapper/enc-loop" successfully created. # vgextend LOOP_VG /dev/mapper/enc-loop Volume group "LOOP_VG" successfully extended # pvs PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 196.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 492.00m # pvmove /dev/loop0 /dev/mapper/enc-loop /dev/loop0: Moved: 30.67% /dev/loop0: Moved: 100.00% # pvs /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it. -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 25/02/19 16:33, Ingo Franzki wrote: we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost. Hello, your message is interesting. And also this thread: https://www.redhat.com/archives/linux-lvm/2019-February/msg2.html But I'd like to know if I understood correctly. Should I care about the physical disk size when I use LVM? Mixing disk with different sector size (512b and 4k) is dangerous? Your message and others from the other thread, seems to say that LVM doesn't handle correctly that situation and that if I pvmove data between a 512b disk and a 4k disk (or viceversa), it will lead to a massive filesystem corruption. If I understood correctly, the problem that you described looks unrelated to encrypted volume and was only exacerbated by that. Right? Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 27.02.2019 01:00, Cesare Leonardi wrote: > On 25/02/19 16:33, Ingo Franzki wrote: >> we just encountered an error when using LVM's pvmove command to move the >> data from an un-encrypted LVM physical volume onto an encrypted volume. >> After the pvmove has completed, the file system on the logical volume that >> resides on the moved physical volumes is corrupted and all data on this LV >> is lost. > > Hello, your message is interesting. And also this thread: > https://www.redhat.com/archives/linux-lvm/2019-February/msg2.html > > But I'd like to know if I understood correctly. > Should I care about the physical disk size when I use LVM? Mixing disk with > different sector size (512b and 4k) is dangerous? As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous. Creating new LVs and thus new file systems on mixed configurations seem to be OK. > > Your message and others from the other thread, seems to say that LVM doesn't > handle correctly that situation and that if I pvmove data between a 512b disk > and a 4k disk (or viceversa), it will lead to a massive filesystem > corruption. If I understood correctly, the problem that you described looks > unrelated to encrypted volume and was only exacerbated by that. Right? Moving from 512 to 4096 seems to cause FS corruption, moving from 4096 to 512 does not. So I guess its only a problem when moving to larger physical block sizes. And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV. E.g. SCSI disks exist in 512 bytes block size and 4096 bytes block size. Or s390-DASDs which always have 4096 bytes blocks. We just encountered it using encrypted disks with the sector-size option to increase encryption performance. So actually we want people to use larger sector sizes, but this seems to cause problems with LVM. The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements. > > Cesare. > > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Wed, 27 Feb 2019, Ingo Franzki wrote: The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements. The losetup command has a --sector-size option that sets the logical sector size. I wonder if that is sufficient to reproduce the problem. -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 27.02.2019 15:59, Stuart D. Gathman wrote: > On Wed, 27 Feb 2019, Ingo Franzki wrote: > >> The good thing about the example with encrypted volumes on loopback >> devices is that you can reproduce the problem on any platform, without >> having certain hardware requirements. > > The losetup command has a --sector-size option that sets the logical > sector size. I wonder if that is sufficient to reproduce the problem. > Yes that should work: # losetup -fP loopbackfile.img --sector-size 4096 # blockdev --getpbsz /dev/loop0 4096 -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 27/02/19 09:49, Ingo Franzki wrote: As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous. Creating new LVs and thus new file systems on mixed configurations seem to be OK. [...] And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV. Thank you Ingo for the precious informations you are giving here. Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms: Sector size (logical/physical): 512 bytes / 4096 bytes What you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte. Do you agree or do you think it would be better to test this specific setup? Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Thu, 28 Feb 2019, Cesare Leonardi wrote: Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms: Sector size (logical/physical): 512 bytes / 4096 bytes What you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte. Do you agree or do you think it would be better to test this specific setup? I would definitely test it, using the same test script that reproduces the problem with loopback devices. That said, I believe you are right - it should definitely work. Most of my drives are 512/4096 logical/phys. If you actually write a single 512 byte sector, however, the disk firmware will have to do a read/modify/write cycle - which can tank performance. hdparm will report logical and physical sector size - but there doesn't seem to be an option to set logical sectory size. There really is no need once you already support a smaller logical sector size, as the performance hit can be avoided by aligned filesystems with 4k+ block size (most modern filesystems). Once I encountered a bug in drive firmware where the R/M/W did not work correctly with certain read/write patterns (involving unaligned multi sector writes). I do not wish that on anyone. (don't worry, that drive model is long gone...). -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 28.02.2019 02:31, Cesare Leonardi wrote: > On 27/02/19 09:49, Ingo Franzki wrote: >> As far as I can tell: Yes if you pvmove data around or lvextend an LV onto >> another PV with a larger physical block size that is dangerous. >> Creating new LVs and thus new file systems on mixed configurations seem to >> be OK. > > [...] > >> And yes, its unrelated to encrypted volumes, it can happen with any block >> device of different physical block sizes that you use as PV. > > Thank you Ingo for the precious informations you are giving here. > > Not to be pedantic, but what do you mean with physical block? Because with > modern disks the term is not always clear. Let's take a mechanical disk with > 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk > will refer to it with these terms: > Sector size (logical/physical): 512 bytes / 4096 bytes > > What you are referring as physical size is actually the logical size reported > by fdisk, right? And if it's correct, I guess that should be safe to add the > above disk with 512e sectors to an LVM storage composed only by disks with > real 512 byte sectors. I expect that from the LVM point of view this should > not be even considered a mixed sector size setup, even if the real physical > sector size of the added disk is 4096 byte. > > Do you agree or do you think it would be better to test this specific setup? Well, there are the following 2 commands: Get physical block size: blockdev --getpbsz Get logical block size: blockdev --getbsz Filesystems seem to care about the physical block size only, not the logical block size. So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful... > > Cesare. > > ___ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
> > Well, there are the following 2 commands: > > Get physical block size: > blockdev --getpbsz > Get logical block size: > blockdev --getbsz > > Filesystems seem to care about the physical block size only, not the logical > block size. > > So as soon as you have PVs with different physical block sizes (as reported > by blockdev --getpbsz) I would be very careful... Hello everybody. Maybe, I don’t understand what do you mean. What the logical block size mean? But on my machines(CentOS7), this utility get me the strange results (output reduced): smartctl -i /dev/sda; blockdev --getbsz --getpbsz /dev/sda Device Model: INTEL SSDSC2KB480G8 User Capacity:480,103,981,056 bytes [480 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate:Solid State Device 4096 4096 smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb Device Model: HGST HUS722T2TALA604 User Capacity:2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate:7200 rpm Form Factor: 3.5 inches 4096 512 As you see “–getbsz” forever 4096. But I think it must be forever 512. What does it mean? Thank you. Ilia. smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 28.02.2019 10:48, Ilia Zykov wrote: >> >> Well, there are the following 2 commands: >> >> Get physical block size: >> blockdev --getpbsz >> Get logical block size: >> blockdev --getbsz >> >> Filesystems seem to care about the physical block size only, not the logical >> block size. >> >> So as soon as you have PVs with different physical block sizes (as reported >> by blockdev --getpbsz) I would be very careful... > > Hello everybody. > Maybe, I don’t understand what do you mean. What the logical block size > mean? But on my machines(CentOS7), this utility get me the strange > results (output reduced): > > smartctl -i /dev/sda; blockdev --getbsz --getpbsz /dev/sda > Device Model: INTEL SSDSC2KB480G8 > User Capacity:480,103,981,056 bytes [480 GB] > Sector Sizes: 512 bytes logical, 4096 bytes physical > Rotation Rate:Solid State Device > 4096 > 4096 > > smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb > Device Model: HGST HUS722T2TALA604 > User Capacity:2,000,398,934,016 bytes [2.00 TB] > Sector Size: 512 bytes logical/physical > Rotation Rate:7200 rpm > Form Factor: 3.5 inches > 4096 > 512 > > As you see “–getbsz” forever 4096. I also see logical block size to be 4096 for all devices on my system. > But I think it must be forever 512. > What does it mean? I have seen the following description about logical and physical block sizes somewhere in the internet: "Logical block sizes are the units used by the 'kernel' for read/write operations. Physical block sizes are the units which 'disk controllers' use for read/write operations." For the problem mentioned in this thread, the physical block size is what you are looking for. > > Thank you. > Ilia. > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
>> >> smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb >> Device Model: HGST HUS722T2TALA604 >> User Capacity:2,000,398,934,016 bytes [2.00 TB] >> Sector Size: 512 bytes logical/physical >> Rotation Rate:7200 rpm >> Form Factor: 3.5 inches >> 4096 >> 512 >> >> As you see “–getbsz” forever 4096. > I also see logical block size to be 4096 for all devices on my system. >> But I think it must be forever 512. >> What does it mean? > I have seen the following description about logical and physical block sizes > somewhere in the internet: > "Logical block sizes are the units used by the 'kernel' for read/write > operations. Kernel can but usually does not want, because reduce performance. > Physical block sizes are the units which 'disk controllers' use for > read/write operations." Not disk controller on the motherboard, but controller inside disk. We don't have access to it. > > For the problem mentioned in this thread, the physical block size is what you > are looking for. >> I think it is BUG in the "blockdev(util-linux)". My question was: Can this error(or similar) be related to a problem in pvmove? smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
> >> >> For the problem mentioned in this thread, the physical block size is what >> you are looking for. >>> > > I think it is BUG in the "blockdev(util-linux)". Tt's not a bug it's a feature :O https://bugzilla.redhat.com/show_bug.cgi?id=1684078 > My question was: > > Can this error(or similar) be related to a problem in pvmove? > > My question now: Can this feature(or similar) be related to a problem in pvmove? Thank you. smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
> Discarding device blocks: done > Creating filesystem with 307200 1k blocks and 76912 inodes > .. > # pvs > /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument > /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument > /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument > /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument > PV VG Fmt Attr PSize PFree > /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m > /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m > > In case the filesystem of the logical volume is not mounted at the time of > pvmove, it gets corrupted anyway, but you only see errors when trying to > mount it. > It's because you FS had 1k blocks. New device can't read with block 1k. If you plan pvmove on device with 4k. Maybe you need make fs with: "mkfs.ext4 -b 4096" See comments: https://bugzilla.redhat.com/show_bug.cgi?id=1684078 smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 28.02.2019 15:36, Ilia Zykov wrote: >> Discarding device blocks: done >> Creating filesystem with 307200 1k blocks and 76912 inodes >> .. >> # pvs >> /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument >> /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument >> /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument >> /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument >> PV VG Fmt Attr PSize PFree >> /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m >> /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m >> >> In case the filesystem of the logical volume is not mounted at the time of >> pvmove, it gets corrupted anyway, but you only see errors when trying to >> mount it. >> > > It's because you FS had 1k blocks. > New device can't read with block 1k. > If you plan pvmove on device with 4k. Maybe you need make fs with: > "mkfs.ext4 -b 40 The beauty of pvmove is that it works on existing LVs with filesystems on it. So I might have filesystems on LVs that used the default block size that I want to move via pvmove. At the time the file system was created (possibly may years ago), I did not know that I would ever move it to a device with a larger block size. If I have to create a new filesystem after the move, then pvmove is useless I then could just create a new LV on the desired PV(s). > > See comments: > https://bugzilla.redhat.com/show_bug.cgi?id=1684078 > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
> At the time the file system was created (possibly may years ago), I did not > know that I would ever move it to a device with a larger block size. > For this purpose all 4k disks have logical sector size 512. Don't look at "blockdev --getbsz" it's not property of physical(real) device. smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 28/02/19 09:41, Ingo Franzki wrote: Well, there are the following 2 commands: Get physical block size: blockdev --getpbsz Get logical block size: blockdev --getbsz I didn't know the blockdev command and, to recap, we have: --getpbsz: physical sector size --getss: logical sector size --getbsz: blocksize used internally by the kernel getpbsz/getss correspond to physical/logical sector size reported by fdisk, smartctl, etc. Filesystems seem to care about the physical block size only, not the logical block size. So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful... I've done the test suggested by Stuart and it seems to contradict this. I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test. Here is what I've done. /dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size # blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 # blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096 # pvcreate /dev/sdb4 # vgcreate vgtest /dev/sdb4 # lvcreate -L 1G vgtest /dev/sdb4 # mkfs.ext4 /dev/mapper/vgtest-lvol0 # mkdir /media/test # mount /dev/mapper/vgtest-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest-lvol0 Filesystem created and no error on it. Now the disk with different physical size: # pvcreate /dev/sdc1 # vgextend vgtest /dev/sdc1 # pvmove /dev/sdb4 /dev/sdc1 # fsck.ext4 -f /dev/mapper/vgtest-lvol0 # mount /dev/mapper/vgtest-lvol0 /media/test # ls /media/test/ The fsck command reports no errors, the filesystem is mountable and the data is there. Looks like physical sector size didn't matter here. Or I'm missing something? Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Fri, 1 Mar 2019, Cesare Leonardi wrote: I've done the test suggested by Stuart and it seems to contradict this. I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test. Use a loopback device with logical block size set to 4096 to confirm that your test does detect corruption (using the same LV, filesystem, data). I believe by "physical sector", the original reporter means logical, as he was using an encrypted block device that was virtual - there was no "physical" sector size. It was "physical" as far as the file system was concerned - where "physical" means "the next layer down". Indeed, even the rotating disk drives make the physical sector size invisible except to performance tests. SSD drives have a "sector" size of 128k or 256k - the erase block, and performance improves when aligned to that. -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 01.03.2019 04:41, Stuart D. Gathman wrote: > On Fri, 1 Mar 2019, Cesare Leonardi wrote: > >> I've done the test suggested by Stuart and it seems to contradict this. >> I have pvmoved data from a 512/512 (logical/physical) disk to a newly added >> 512/4096 disk but I had no data corruption. Unfortunately I haven't any >> native 4k disk to repeat the same test. > > Use a loopback device with logical block size set to 4096 to confirm > that your test does detect corruption (using the same LV, filesystem, > data). > > I believe by "physical sector", the original reporter means logical, > as he was using an encrypted block device that was virtual - there > was no "physical" sector size. It was "physical" as far as the > file system was concerned - where "physical" means "the next layer > down". Well, let me cite from https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html from Ondrej Kozina which is also referenced in my original post: "dm-crypt advertise itself as a block device with physical sector size *at least* equal to encryption sector size. Traditionally it's been only 512B. So classical dm-crypt mapped over device with phys. sector size = 512B has no impact. If you mapped dm-crypt over block device with native physical sector size = 4096 you got dm-crypt exposing same limits as underlying block device. Again no problem. Just internally dm-crypt performed encryption on 512B blocks, but it had no impact on exposed limits. But things get a bit different with encryption sector size > 512B. If you map dm-crypt with encryption sector size set to 4096B over block device with physical sector size = 512B, dm-crypt must increase device limits to 4096. Because when it does encryption it must be aligned to 4096 bytes (and same wrt minimal i/o size)." > > Indeed, even the rotating disk drives make the physical sector size > invisible except to performance tests. SSD drives have a "sector" size > of 128k or 256k - the erase block, and performance improves when aligned > to that. > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 01.03.2019 02:24, Cesare Leonardi wrote: > On 28/02/19 09:41, Ingo Franzki wrote: >> Well, there are the following 2 commands: >> >> Get physical block size: >> blockdev --getpbsz >> Get logical block size: >> blockdev --getbsz > > I didn't know the blockdev command and, to recap, we have: > --getpbsz: physical sector size > --getss: logical sector size > --getbsz: blocksize used internally by the kernel > > getpbsz/getss correspond to physical/logical sector size reported by fdisk, > smartctl, etc. > >> Filesystems seem to care about the physical block size only, not the logical >> block size. >> >> So as soon as you have PVs with different physical block sizes (as reported >> by blockdev --getpbsz) I would be very careful... > > I've done the test suggested by Stuart and it seems to contradict this. > I have pvmoved data from a 512/512 (logical/physical) disk to a newly added > 512/4096 disk but I had no data corruption. Unfortunately I haven't any > native 4k disk to repeat the same test. Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. Once the FS uses 4K block anyway you wont see the problem. Use tune2fs -l after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device. > > Here is what I've done. > /dev/sdb: SSD with 512/512 sector size > /dev/sdc: mechanical disk with 512/4096 sector size > > # blockdev -v --getss --getpbsz --getbsz /dev/sdb > get logical block (sector) size: 512 > get physical block (sector) size: 512 > get blocksize: 4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/sdc > get logical block (sector) size: 512 > get physical block (sector) size: 4096 > get blocksize: 4096 > > # pvcreate /dev/sdb4 > # vgcreate vgtest /dev/sdb4 > # lvcreate -L 1G vgtest /dev/sdb4 > # mkfs.ext4 /dev/mapper/vgtest-lvol0 > # mkdir /media/test > # mount /dev/mapper/vgtest-lvol0 /media/test > # cp -a SOMEDATA /media/test/ > # umount /media/test > # fsck.ext4 -f /dev/mapper/vgtest-lvol0 > > Filesystem created and no error on it. Now the disk with different physical > size: > > # pvcreate /dev/sdc1 > # vgextend vgtest /dev/sdc1 > # pvmove /dev/sdb4 /dev/sdc1 > # fsck.ext4 -f /dev/mapper/vgtest-lvol0 > # mount /dev/mapper/vgtest-lvol0 /media/test > # ls /media/test/ > > The fsck command reports no errors, the filesystem is mountable and the data > is there. > > Looks like physical sector size didn't matter here. Or I'm missing something? > > Cesare. > > ___ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
Hello Ingo, I've made several tests but I was unable to trigger any filesystem corruption. Maybe the trouble you encountered are specific to encrypted device? Yesterday and today I've used: Debian unstable kernel 4.19.20 lvm2 2.03.02 e2fsprogs 1.44.5 On 01/03/19 09:05, Ingo Franzki wrote: Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. Once the FS uses 4K block anyway you wont see the problem. Use tune2fs -l after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device. I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem I've used. I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a +512M partition, but use 1024 for +500M. As suggested by Stuart, I also made a test using a 4k loop device and pvmoving the LV into it. As you expected, no data corruption. To do it I've recreated the same setup ad yesterday: /dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on it. Then: # fallocate -l 10G testdisk.img # losetup -f -L -P -b 4096 testdisk.img # pvcreate /dev/loop0 # vgextend vgtest /dev/loop0 # pvmove /dev/sdb4 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest-lvol0 While I was there, out of curiosity, I've created an ext4 filesystem on a <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 disk to 512/4096, then to the 4096/4096 loop device. New partitions and a new VG was used for that. The setup: /dev/sdb5: 512/512 /dev/sdc2: 512/4096 /dev/loop0 4096/4096 # blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 # blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096 # blockdev -v --getss --getpbsz --getbsz /dev/loop0 get logical block (sector) size: 4096 get physical block (sector) size: 4096 get blocksize: 4096 # pvcreate /dev/sdb5 # vgcreate vgtest2 /dev/sdb5 # lvcreate -L 400M vgtest2 /dev/sdb5 # mkfs.ext4 /dev/mapper/vgtest2-lvol0 # tune2fs -l /dev/mapper/vgtest2-lvol0 [...] Block size: 1024 [...] # mount /dev/mapper/vgtest2-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Now I've moved data from the 512/512 to the 512/4096 disk: # pvcreate /dev/sdc2 # vgextend vgtest2 /dev/sdc2 # pvmove /dev/sdb5 /dev/sdc2 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 No error reported. Now I've moved data to the 4096/4096 loop device: # pvcreate /dev/loop0 # vgextend vgtest2 /dev/loop0 # pvmove /dev/sdc2 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Still no data corruption. Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 2/27/2019 9:05 AM, Ingo Franzki wrote: > Yes that should work: > # losetup -fP loopbackfile.img --sector-size 4096 > # blockdev --getpbsz /dev/loop0 > 4096 > - Something I noticed that is troublesome. When I first got my 4K sectory size disks, one of the numbers in the kernel listed it as 4K, now ~4-5 years later, that difference no longer appears in the kernel! To me, this seems a bit unwise -- as even though these disks are "512_e_" (a 512 byte sector size can be emulated but writes will usually or often take a minimum of 2 revolutions so that the rest of the 4k sector can be read, have 512-bytes updated, then write the whole 4k sector size out. In otherwords -- hideous performance on writes. Now the RAID controller that manages this disk also has info on the disks -- and there it correctly shows the 4k sector size. So am wondering what in the kernel "broke"(?) to disable detection & display of its native 4k sector size. Anyway, I don't think it should cause problems unless I reformat or move the partitions, but it would be nice to know the various disk software knows not to try anything smaller than a 4k sector size. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
>> >> smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb >> Device Model: HGST HUS722T2TALA604 >> User Capacity:2,000,398,934,016 bytes [2.00 TB] >> Sector Size: 512 bytes logical/physical >> Rotation Rate:7200 rpm >> Form Factor: 3.5 inches >> 4096 >> 512 >> >> As you see “–getbsz” forever 4096. > I also see logical block size to be 4096 for all devices on my system. >> But I think it must be forever 512. >> What does it mean? > I have seen the following description about logical and physical block sizes > somewhere in the internet: > "Logical block sizes are the units used by the 'kernel' for read/write > operations. Kernel can but usually does not want, because reduce performance. > Physical block sizes are the units which 'disk controllers' use for > read/write operations." Not disk controller on the motherboard, but controller inside disk. We don't have access to it. > > For the problem mentioned in this thread, the physical block size is what you > are looking for. >> I think it is BUG in the "blockdev". My question was: Can this error(or similar) be related to a problem in pvmove? smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Sat, Mar 2, 2019 at 3:38 AM Cesare Leonardi wrote: > Hello Ingo, I've made several tests but I was unable to trigger any > filesystem corruption. Maybe the trouble you encountered are specific to > encrypted device? > > Yesterday and today I've used: > Debian unstable > kernel 4.19.20 > lvm2 2.03.02 > e2fsprogs 1.44.5 > > On 01/03/19 09:05, Ingo Franzki wrote: > > Hmm, maybe the size of the volume plays a role as Bernd has pointed out. > ext4 may use -b 4K by default on larger devices. > > Once the FS uses 4K block anyway you wont see the problem. > > > > Use tune2fs -l after you created the file system and check if > it is using 4K blocks on your 512/512 device. If so, then you won't see the > problem when moved to a 4K block size device. > > I confirm that tune2fs reports 4096 block size for the 1 GB ext4 > filesystem I've used. > I've also verified what Bernd said: mkfs.ext4 still use 4096 block size > for a +512M partition, but use 1024 for +500M. > > As suggested by Stuart, I also made a test using a 4k loop device and > pvmoving the LV into it. As you expected, no data corruption. > To do it I've recreated the same setup ad yesterday: > /dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on > it. Then: > # fallocate -l 10G testdisk.img > # losetup -f -L -P -b 4096 testdisk.img > # pvcreate /dev/loop0 > # vgextend vgtest /dev/loop0 > # pvmove /dev/sdb4 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest-lvol0 > > While I was there, out of curiosity, I've created an ext4 filesystem on > a <500MB LV (block size = 1024) and I've tried pvmoving data from the > 512/512 disk to 512/4096, then to the 4096/4096 loop device. > New partitions and a new VG was used for that. > > The setup: > /dev/sdb5: 512/512 > /dev/sdc2: 512/4096 > /dev/loop0 4096/4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/sdb > get logical block (sector) size: 512 > get physical block (sector) size: 512 > get blocksize: 4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/sdc > get logical block (sector) size: 512 > get physical block (sector) size: 4096 > get blocksize: 4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/loop0 > get logical block (sector) size: 4096 > get physical block (sector) size: 4096 > get blocksize: 4096 > > # pvcreate /dev/sdb5 > # vgcreate vgtest2 /dev/sdb5 > # lvcreate -L 400M vgtest2 /dev/sdb5 > # mkfs.ext4 /dev/mapper/vgtest2-lvol0 > > # tune2fs -l /dev/mapper/vgtest2-lvol0 > [...] > Block size: 1024 > [...] > > # mount /dev/mapper/vgtest2-lvol0 /media/test > # cp -a SOMEDATA /media/test/ > # umount /media/test > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 > > Now I've moved data from the 512/512 to the 512/4096 disk: > # pvcreate /dev/sdc2 > # vgextend vgtest2 /dev/sdc2 > # pvmove /dev/sdb5 /dev/sdc2 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 > > No error reported. > Did you try to mount the lv after the pvmove? > Now I've moved data to the 4096/4096 loop device: > # pvcreate /dev/loop0 > # vgextend vgtest2 /dev/loop0 > # pvmove /dev/sdc2 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 > > Still no data corruption. > I can reproduce this without moving data, just by extending vg with 4k device, and then extending lv to use both devices. Here is what I tested: # truncate -s 500m disk1 # truncate -s 500m disk2 # losetup -f disk1 --sector-size 512 --show /dev/loop2 # losetup -f disk2 --sector-size 4096 --show /dev/loop3 # pvcreate /dev/loop2 Physical volume "/dev/loop2" successfully created. # pvcreate /dev/loop3 Physical volume "/dev/loop3" successfully created. # vgcreate test /dev/loop2 Volume group "test" successfully created # lvcreate -L400m -n lv1 test Logical volume "lv1" created. # mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512agcount=4, agsize=25600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 # mkdir /tmp/mnt # mount /dev/test/lv1 /tmp/mnt # vgextend test /dev/loop3 Volume group "test" successfully extended # lvextend -L+400m test/lv1 Size of logical volume test/lv1 changed from 400.00 MiB (100 extents) to 800.00 MiB (200 extents). Logical volume test/lv1 successfully resized. # umount /tmp/mnt # mount /dev/test/lv1 /tmp/mnt mount: /tmp/mnt: mount(2) system call failed: Function not implemented. >From journalctl: Mar 02 21:52:53 lean.local kernel: XFS (dm-7): Unmounting Filesystem Mar 02 21:53:01 lean.local kernel: XFS (dm-7): device supports 4096 byte sectors (not 512) I also tried the same
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 02.03.2019 02:36, Cesare Leonardi wrote: > Hello Ingo, I've made several tests but I was unable to trigger any > filesystem corruption. Maybe the trouble you encountered are specific to > encrypted device? > > Yesterday and today I've used: > Debian unstable > kernel 4.19.20 > lvm2 2.03.02 > e2fsprogs 1.44.5 > > On 01/03/19 09:05, Ingo Franzki wrote: >> Hmm, maybe the size of the volume plays a role as Bernd has pointed out. >> ext4 may use -b 4K by default on larger devices. >> Once the FS uses 4K block anyway you wont see the problem. >> >> Use tune2fs -l after you created the file system and check if it >> is using 4K blocks on your 512/512 device. If so, then you won't see the >> problem when moved to a 4K block size device. > > I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem > I've used. > I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a > +512M partition, but use 1024 for +500M. > > As suggested by Stuart, I also made a test using a 4k loop device and > pvmoving the LV into it. As you expected, no data corruption. > To do it I've recreated the same setup ad yesterday: /dev/mapper/vgtest-lvol0 > on /dev/sdb4, a 512/512 disk, with some data on it. Then: > # fallocate -l 10G testdisk.img > # losetup -f -L -P -b 4096 testdisk.img > # pvcreate /dev/loop0 > # vgextend vgtest /dev/loop0 > # pvmove /dev/sdb4 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest-lvol0 > > While I was there, out of curiosity, I've created an ext4 filesystem on a > <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 > disk to 512/4096, then to the 4096/4096 loop device. > New partitions and a new VG was used for that. > > The setup: > /dev/sdb5: 512/512 > /dev/sdc2: 512/4096 > /dev/loop0 4096/4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/sdb > get logical block (sector) size: 512 > get physical block (sector) size: 512 > get blocksize: 4096 You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on. Not sure if this makes a difference > > # blockdev -v --getss --getpbsz --getbsz /dev/sdc > get logical block (sector) size: 512 > get physical block (sector) size: 4096 > get blocksize: 4096 Again, you display the physical block size of /dev/sdc here, but you use /dev/sdc2 later on. > > # blockdev -v --getss --getpbsz --getbsz /dev/loop0 > get logical block (sector) size: 4096 > get physical block (sector) size: 4096 > get blocksize: 4096 > > # pvcreate /dev/sdb5 > # vgcreate vgtest2 /dev/sdb5 > # lvcreate -L 400M vgtest2 /dev/sdb5 > # mkfs.ext4 /dev/mapper/vgtest2-lvol0 > > # tune2fs -l /dev/mapper/vgtest2-lvol0 > [...] > Block size: 1024 > [...] > > # mount /dev/mapper/vgtest2-lvol0 /media/test > # cp -a SOMEDATA /media/test/ > # umount /media/test > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 > > Now I've moved data from the 512/512 to the 512/4096 disk: > # pvcreate /dev/sdc2 > # vgextend vgtest2 /dev/sdc2 > # pvmove /dev/sdb5 /dev/sdc2 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Please note that fsck.ext4 does not seem to detect this kind of corruption. In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore... Do a 'pvs' command here, this should show some error messages. > > No error reported. > Now I've moved data to the 4096/4096 loop device: > # pvcreate /dev/loop0 > # vgextend vgtest2 /dev/loop0 > # pvmove /dev/sdc2 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Again, fsck did not show the corruption for me, but pvs did show error messages. > > Still no data corruption. > > Cesare. > > ___ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 04/03/19 10:12, Ingo Franzki wrote: # blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on. Not sure if this makes a difference I thought that was the right thing to do, as they are disk parameters. At least the first two, for the last I'm not sure. However the output looks the same: # blockdev -v --getss --getpbsz --getbsz /dev/sdb5 get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 # blockdev -v --getss --getpbsz --getbsz /dev/sdc2 get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096 Please note that fsck.ext4 does not seem to detect this kind of corruption. In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore... Do a 'pvs' command here, this should show some error messages. Uh, I didn't really expect that such corruption could pass unoticed an fsck.ext4 check. During my tests, initially I surely tried to mount the filesystem and I did ls on it but it's possible that after some steps I only trusted fsck. Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size. Here is what I've tested: /dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size /dev/loop0: emulated disk with 4096/4096 sector size TEST 1 VG vgtest1: /dev/sdb4 /dev/sdc1 /dev/loop0p1 LV vgtest1-lvol0: filesystem ext4 with 4096 block size pvmove ext4-4096: - from /dev/sdb4 (512/512) to /dev/sdc1 (512/4096): ok - from /dev/sdc1 (512/4096) to /dev/loop0p1 (4096/4096): ok TEST 2 VG vgtest2: /dev/sdb5 /dev/sdc2 /dev/loop0p2 LV vgtest2-lvol0: filesystem ext4 with 1024 block size pvmove ext4-1024: - from /dev/sdb5 (512/512) to /dev/sdc2 (512/4096): ok - from /dev/sdc2 (512/4096) to /dev/loop0p2 (4096/4096): fail Here the outputs of the failed test: === # pvmove /dev/sdc2 /dev/loop0p2 /dev/sdc2: Moved: 9,00% /dev/sdc2: Moved: 100,00% # mount /dev/mapper/vgtest2-lvol0 /media/test/ mount: /media/test: wrong fs type, bad option, bad superblock on /dev/mapper/vgtest2-lvol0, missing codepage or helper program, or other error. # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 e2fsck 1.44.5 (15-Dec-2018) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/vgtest2-lvol0: 35/102400 files (17.1% non-contiguous), 304877/409600 blocks === The error happened where you guys expected. And also for me fsck showed no errors. But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again: # pvmove /dev/loop0p2 /dev/sdc2 /dev/loop0p2: Moved: 1,00% /dev/loop0p2: Moved: 100,00% # mount /dev/mapper/vgtest2-lvol0 /media/test/ # ls /media/test/ epson hp kerio lost+found And also notice that the pvmove that generated the unreadable filesystem starts with an unusual high percentage (9%). In all other test it start from 0% or a small number near 1%. This happened in more that one case. Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 02/03/19 21:25, Nir Soffer wrote: # mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Has the problem here the same root as for ext4? I guess sectsz should be >=4096 to avoid troubles, isn't it? Just to draw some conlusion, could we say that currently, if we are going to move data around with LVM, it's better to check that the filesystem is using a block size >= than "blockdev --getbsz DESTINATIONDEVICE"? At least with ext4 and xfs. Something that couldn't be true with really small devices (< 500 MB). Is there already an open bug regarding the problem discussed in this thread? Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Mon, 4 Mar 2019, Cesare Leonardi wrote: Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size. ... The error happened where you guys expected. And also for me fsck showed no errors. But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again: ... THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device. I suspect it could also happen with an unaligned filesystem on a 4096 device. -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 05.03.2019 01:12, Stuart D. Gathman wrote: > On Mon, 4 Mar 2019, Cesare Leonardi wrote: > >> Today I repeated all the tests and indeed in one case the mount failed: >> after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV >> ext4 using 1024 block size. > ... >> The error happened where you guys expected. And also for me fsck showed no >> errors. >> >> But doesn't look like a filesystem corruption: if you pvmove back the data, >> it will become readable again: > ... > > THAT is a crucial observation. It's not an LVM bug, but the filesystem > trying to read 1024 bytes on a 4096 device. Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device. The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case). I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble. > I suspect it could also > happen with an unaligned filesystem on a 4096 device. > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 05.03.2019 00:22, Nir Soffer wrote: > On Tue, Mar 5, 2019 at 12:45 AM Cesare Leonardi wrote: > >> On 02/03/19 21:25, Nir Soffer wrote: >>> # mkfs.xfs /dev/test/lv1 >>> meta-data=/dev/test/lv1 isize=512agcount=4, agsize=25600 >> blks >>> = sectsz=512 attr=2, projid32bit=1 >>> = crc=1finobt=1, sparse=0, >>> rmapbt=0, reflink=0 >>> data = bsize=4096 blocks=102400, imaxpct=25 >>> = sunit=0 swidth=0 blks >>> naming =version 2 bsize=4096 ascii-ci=0 ftype=1 >>> log =internal log bsize=4096 blocks=855, version=2 >>> = sectsz=512 sunit=0 blks, lazy-count=1 >>> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> Has the problem here the same root as for ext4? I guess sectsz should be >> >=4096 to avoid troubles, isn't it? >> >> Just to draw some conlusion, could we say that currently, if we are >> going to move data around with LVM, it's better to check that the >> filesystem is using a block size >= than "blockdev --getbsz >> DESTINATIONDEVICE"? At least with ext4 and xfs. >> >> Something that couldn't be true with really small devices (< 500 MB). >> >> Is there already an open bug regarding the problem discussed in this >> thread? >> > > There is this bug about lvextend: > https://bugzilla.redhat.com/1669751 > > And this old bug from 2011, discussing mixing PVs with different block size. > Comment 2 is very clear about this issue: > https://bugzilla.redhat.com/732980#c2 I don't have access to that one, can you cite comment 2 ? > > Nir > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
Hello. >> THAT is a crucial observation. It's not an LVM bug, but the filesystem >> trying to read 1024 bytes on a 4096 device. > Yes that's probably the reason. Nevertheless, its not really the FS's fault, > since it was moved by LVM to a 4069 device. > The FS does not know anything about the move, so it reads in the block size > it was created with (1024 in this case). > > I still think LVM should prevent one from mixing devices with different > physical block sizes, or at least warn when pvmoving or lvextending onto a PV > with a larger block size, since this can cause trouble. > In this case, "dd" tool and others should prevent too. Because after: dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;) /dev/DiskWith512block has ext4 fs with 1k block. P.S. LVM,dd .. are low level tools and doesn't know about hi level anything. And in the your case and others cases can't know. You should test(if you need) the block size with other tools before moving or copying. Not a lvm bug. Thank you. smime.p7s Description: S/MIME Cryptographic Signature ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 05.03.2019 10:29, Ilia Zykov wrote: > Hello. > >>> THAT is a crucial observation. It's not an LVM bug, but the filesystem >>> trying to read 1024 bytes on a 4096 device. >> Yes that's probably the reason. Nevertheless, its not really the FS's fault, >> since it was moved by LVM to a 4069 device. >> The FS does not know anything about the move, so it reads in the block size >> it was created with (1024 in this case). >> >> I still think LVM should prevent one from mixing devices with different >> physical block sizes, or at least warn when pvmoving or lvextending onto a >> PV with a larger block size, since this can cause trouble. >> > > In this case, "dd" tool and others should prevent too. Well, no, its LVM's pvmove who moves the data from a 512 block size to 4096 block size device. So its not dd that's causing the problem, but pvmove. > > Because after: > > dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock > > You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;) > /dev/DiskWith512block has ext4 fs with 1k block. > > P.S. > LVM,dd .. are low level tools and doesn't know about hi level anything. > And in the your case and others cases can't know. You should test(if you > need) the block size with other tools before moving or copying. > Not a lvm bug. Well, maybe not a bug, but I would still like to see that LVM's pvmove/lvextend and/or vgextend issues at least a warning when you have mixed block sizes. LVM knows the block sizes of the PVs it manages and it also knows when it changes the block size of an LV due to a pvmove or lvextend. So it can issue a warning (better a confirmation prompt) when this happens. > Thank you. > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Tue, Mar 05, 2019 at 06:29:31PM +0200, Nir Soffer wrote: > I don't this way of thinking is useful. If we go in this way, then write() > should not > let you write data, and later maybe the disk controller should avoid this? > > LVM is not a low level tool like dd. It is high level tool for managing > device mapper, > and providing high level tools to create user level abstractions. We can > expect it > to prevent system administrator from doing the wrong thing. > > Maybe LVM should let you mix PVs with different logical block size, but it > should > require --force. > > David, what do you think? LVM needs to fix this, your solution sounds like the right one. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Tue, 5 Mar 2019, David Teigland wrote: On Tue, Mar 05, 2019 at 06:29:31PM +0200, Nir Soffer wrote: Maybe LVM should let you mix PVs with different logical block size, but it should require --force. LVM needs to fix this, your solution sounds like the right one. Also, since nearly every modern device device has a physical block size of 4k or more, and even when the logical block size is (emulated) 512, performance degradation occurs with smaller filesystem blocks, then the savvy admin should ensure that all filesystem have a min of 4k block size - except in special circustances. -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Tue, Mar 5, 2019 at 11:30 AM Ilia Zykov wrote: > Hello. > > >> THAT is a crucial observation. It's not an LVM bug, but the filesystem > >> trying to read 1024 bytes on a 4096 device. > > Yes that's probably the reason. Nevertheless, its not really the FS's > fault, since it was moved by LVM to a 4069 device. > > The FS does not know anything about the move, so it reads in the block > size it was created with (1024 in this case). > > > > I still think LVM should prevent one from mixing devices with different > physical block sizes, or at least warn when pvmoving or lvextending onto a > PV with a larger block size, since this can cause trouble. > > > > In this case, "dd" tool and others should prevent too. > > Because after: > > dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock > > You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;) > /dev/DiskWith512block has ext4 fs with 1k block. > > P.S. > LVM,dd .. are low level tools and doesn't know about hi level anything. > And in the your case and others cases can't know. You should test(if you > need) the block size with other tools before moving or copying. > Not a lvm bug. > I don't this way of thinking is useful. If we go in this way, then write() should not let you write data, and later maybe the disk controller should avoid this? LVM is not a low level tool like dd. It is high level tool for managing device mapper, and providing high level tools to create user level abstractions. We can expect it to prevent system administrator from doing the wrong thing. Maybe LVM should let you mix PVs with different logical block size, but it should require --force. David, what do you think? ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Tue, Mar 5, 2019 at 12:45 AM Cesare Leonardi wrote: > On 02/03/19 21:25, Nir Soffer wrote: > > # mkfs.xfs /dev/test/lv1 > > meta-data=/dev/test/lv1 isize=512agcount=4, agsize=25600 > blks > > = sectsz=512 attr=2, projid32bit=1 > > = crc=1finobt=1, sparse=0, > > rmapbt=0, reflink=0 > > data = bsize=4096 blocks=102400, imaxpct=25 > > = sunit=0 swidth=0 blks > > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > > log =internal log bsize=4096 blocks=855, version=2 > > = sectsz=512 sunit=0 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > Has the problem here the same root as for ext4? I guess sectsz should be > >=4096 to avoid troubles, isn't it? > > Just to draw some conlusion, could we say that currently, if we are > going to move data around with LVM, it's better to check that the > filesystem is using a block size >= than "blockdev --getbsz > DESTINATIONDEVICE"? At least with ext4 and xfs. > > Something that couldn't be true with really small devices (< 500 MB). > > Is there already an open bug regarding the problem discussed in this > thread? > There is this bug about lvextend: https://bugzilla.redhat.com/1669751 And this old bug from 2011, discussing mixing PVs with different block size. Comment 2 is very clear about this issue: https://bugzilla.redhat.com/732980#c2 Nir ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/