Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 05.03.2019 01:12, Stuart D. Gathman wrote: > On Mon, 4 Mar 2019, Cesare Leonardi wrote: > >> Today I repeated all the tests and indeed in one case the mount failed: >> after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV >> ext4 using 1024 block size. > ... >> The error happened where you guys expected. And also for me fsck showed no >> errors. >> >> But doesn't look like a filesystem corruption: if you pvmove back the data, >> it will become readable again: > ... > > THAT is a crucial observation. It's not an LVM bug, but the filesystem > trying to read 1024 bytes on a 4096 device. Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device. The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case). I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble. > I suspect it could also > happen with an unaligned filesystem on a 4096 device. > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On Mon, 4 Mar 2019, Cesare Leonardi wrote: Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size. ... The error happened where you guys expected. And also for me fsck showed no errors. But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again: ... THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device. I suspect it could also happen with an unaligned filesystem on a 4096 device. -- Stuart D. Gathman "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 02/03/19 21:25, Nir Soffer wrote: # mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Has the problem here the same root as for ext4? I guess sectsz should be >=4096 to avoid troubles, isn't it? Just to draw some conlusion, could we say that currently, if we are going to move data around with LVM, it's better to check that the filesystem is using a block size >= than "blockdev --getbsz DESTINATIONDEVICE"? At least with ext4 and xfs. Something that couldn't be true with really small devices (< 500 MB). Is there already an open bug regarding the problem discussed in this thread? Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 04/03/19 10:12, Ingo Franzki wrote: # blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on. Not sure if this makes a difference I thought that was the right thing to do, as they are disk parameters. At least the first two, for the last I'm not sure. However the output looks the same: # blockdev -v --getss --getpbsz --getbsz /dev/sdb5 get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096 # blockdev -v --getss --getpbsz --getbsz /dev/sdc2 get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096 Please note that fsck.ext4 does not seem to detect this kind of corruption. In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore... Do a 'pvs' command here, this should show some error messages. Uh, I didn't really expect that such corruption could pass unoticed an fsck.ext4 check. During my tests, initially I surely tried to mount the filesystem and I did ls on it but it's possible that after some steps I only trusted fsck. Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size. Here is what I've tested: /dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size /dev/loop0: emulated disk with 4096/4096 sector size TEST 1 VG vgtest1: /dev/sdb4 /dev/sdc1 /dev/loop0p1 LV vgtest1-lvol0: filesystem ext4 with 4096 block size pvmove ext4-4096: - from /dev/sdb4 (512/512) to /dev/sdc1 (512/4096): ok - from /dev/sdc1 (512/4096) to /dev/loop0p1 (4096/4096): ok TEST 2 VG vgtest2: /dev/sdb5 /dev/sdc2 /dev/loop0p2 LV vgtest2-lvol0: filesystem ext4 with 1024 block size pvmove ext4-1024: - from /dev/sdb5 (512/512) to /dev/sdc2 (512/4096): ok - from /dev/sdc2 (512/4096) to /dev/loop0p2 (4096/4096): fail Here the outputs of the failed test: === # pvmove /dev/sdc2 /dev/loop0p2 /dev/sdc2: Moved: 9,00% /dev/sdc2: Moved: 100,00% # mount /dev/mapper/vgtest2-lvol0 /media/test/ mount: /media/test: wrong fs type, bad option, bad superblock on /dev/mapper/vgtest2-lvol0, missing codepage or helper program, or other error. # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 e2fsck 1.44.5 (15-Dec-2018) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/vgtest2-lvol0: 35/102400 files (17.1% non-contiguous), 304877/409600 blocks === The error happened where you guys expected. And also for me fsck showed no errors. But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again: # pvmove /dev/loop0p2 /dev/sdc2 /dev/loop0p2: Moved: 1,00% /dev/loop0p2: Moved: 100,00% # mount /dev/mapper/vgtest2-lvol0 /media/test/ # ls /media/test/ epson hp kerio lost+found And also notice that the pvmove that generated the unreadable filesystem starts with an unusual high percentage (9%). In all other test it start from 0% or a small number near 1%. This happened in more that one case. Cesare. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
On 02.03.2019 02:36, Cesare Leonardi wrote: > Hello Ingo, I've made several tests but I was unable to trigger any > filesystem corruption. Maybe the trouble you encountered are specific to > encrypted device? > > Yesterday and today I've used: > Debian unstable > kernel 4.19.20 > lvm2 2.03.02 > e2fsprogs 1.44.5 > > On 01/03/19 09:05, Ingo Franzki wrote: >> Hmm, maybe the size of the volume plays a role as Bernd has pointed out. >> ext4 may use -b 4K by default on larger devices. >> Once the FS uses 4K block anyway you wont see the problem. >> >> Use tune2fs -l after you created the file system and check if it >> is using 4K blocks on your 512/512 device. If so, then you won't see the >> problem when moved to a 4K block size device. > > I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem > I've used. > I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a > +512M partition, but use 1024 for +500M. > > As suggested by Stuart, I also made a test using a 4k loop device and > pvmoving the LV into it. As you expected, no data corruption. > To do it I've recreated the same setup ad yesterday: /dev/mapper/vgtest-lvol0 > on /dev/sdb4, a 512/512 disk, with some data on it. Then: > # fallocate -l 10G testdisk.img > # losetup -f -L -P -b 4096 testdisk.img > # pvcreate /dev/loop0 > # vgextend vgtest /dev/loop0 > # pvmove /dev/sdb4 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest-lvol0 > > While I was there, out of curiosity, I've created an ext4 filesystem on a > <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 > disk to 512/4096, then to the 4096/4096 loop device. > New partitions and a new VG was used for that. > > The setup: > /dev/sdb5: 512/512 > /dev/sdc2: 512/4096 > /dev/loop0 4096/4096 > > # blockdev -v --getss --getpbsz --getbsz /dev/sdb > get logical block (sector) size: 512 > get physical block (sector) size: 512 > get blocksize: 4096 You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on. Not sure if this makes a difference > > # blockdev -v --getss --getpbsz --getbsz /dev/sdc > get logical block (sector) size: 512 > get physical block (sector) size: 4096 > get blocksize: 4096 Again, you display the physical block size of /dev/sdc here, but you use /dev/sdc2 later on. > > # blockdev -v --getss --getpbsz --getbsz /dev/loop0 > get logical block (sector) size: 4096 > get physical block (sector) size: 4096 > get blocksize: 4096 > > # pvcreate /dev/sdb5 > # vgcreate vgtest2 /dev/sdb5 > # lvcreate -L 400M vgtest2 /dev/sdb5 > # mkfs.ext4 /dev/mapper/vgtest2-lvol0 > > # tune2fs -l /dev/mapper/vgtest2-lvol0 > [...] > Block size: 1024 > [...] > > # mount /dev/mapper/vgtest2-lvol0 /media/test > # cp -a SOMEDATA /media/test/ > # umount /media/test > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 > > Now I've moved data from the 512/512 to the 512/4096 disk: > # pvcreate /dev/sdc2 > # vgextend vgtest2 /dev/sdc2 > # pvmove /dev/sdb5 /dev/sdc2 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Please note that fsck.ext4 does not seem to detect this kind of corruption. In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore... Do a 'pvs' command here, this should show some error messages. > > No error reported. > Now I've moved data to the 4096/4096 loop device: > # pvcreate /dev/loop0 > # vgextend vgtest2 /dev/loop0 > # pvmove /dev/sdc2 /dev/loop0 > # fsck.ext4 -f /dev/mapper/vgtest2-lvol0 Again, fsck did not show the corruption for me, but pvs did show error messages. > > Still no data corruption. > > Cesare. > > ___ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > -- Ingo Franzki eMail: ifran...@linux.ibm.com Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
