Bug#1063422: [regression 6.1.y] f2fs: invalid zstd compress level: 6
On 2024/2/20 3:54, Salvatore Bonaccorso wrote: Hi, On Mon, Feb 19, 2024 at 10:35:13AM +0800, Chao Yu wrote: On 2024/2/9 4:19, Salvatore Bonaccorso wrote: Hi Jaegeuk Kim, Chao Yu, In Debian the following regression was reported after a Dhya updated to 6.1.76: On Wed, Feb 07, 2024 at 10:43:47PM -0500, Dhya wrote: Package: src:linux Version: 6.1.76-1 Severity: critical Justification: breaks the whole system Dear Maintainer, After upgrade to linux-image-6.1.0-18-amd64 6.1.76-1 F2FS filesystem fails to mount rw. Message in the boot journal: kernel: F2FS-fs (nvme0n1p6): invalid zstd compress level: 6 There was recently an f2fs patch to the 6.1 kernel tree which might be related: https://www.spinics.net/lists/stable-commits/msg329957.html Was able to recover the system by doing: sudo mount -o remount,rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,compress_algorithm=lz4,compress_log_size=2,compress_mode=fs,atgc,discard_unit=block,memory=normal /dev/nvme0n1p6 / under the running bad 6.1.0-18-amd64 kernel, then editing /etc/default/grub: GRUB_DEFAULT="Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-17-amd64" and running 'update-grub' and rebooting to boot the 6.1.0-17-amd64 kernel. The issue is easily reproducible by: # dd if=/dev/zero of=test.img count=100 bs=1M # mkfs.f2fs -f -O compression,extra_attr ./test.img # mount -t f2fs -o compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime ./test.img /mnt resulting in [ 60.789982] F2FS-fs (loop0): invalid zstd compress level: 6 Hi Salvatore, Can you please try below fixes: [PATCH 6.1] f2fs: add helper to check compression level https://lore.kernel.org/linux-f2fs-devel/20240212160530.1017205-1-c...@kernel.org Confirmed that this fixes the reported issue as it was reported to us in Debian in https://bugs.debian.org/1063422 . Thanks a lot! (note just tested with the first commit as it landed in 6.1.78 to confirm the immediate regression). #regzbot fixed-by: cf3d57ad6ff8b566deba3544b9ad3384781fb604 Hi, Thank you for confirmation. Thanks, Regards, Salvatore
Bug#1063422: [regression 6.1.y] f2fs: invalid zstd compress level: 6
On 2024/2/9 4:19, Salvatore Bonaccorso wrote: Hi Jaegeuk Kim, Chao Yu, In Debian the following regression was reported after a Dhya updated to 6.1.76: On Wed, Feb 07, 2024 at 10:43:47PM -0500, Dhya wrote: Package: src:linux Version: 6.1.76-1 Severity: critical Justification: breaks the whole system Dear Maintainer, After upgrade to linux-image-6.1.0-18-amd64 6.1.76-1 F2FS filesystem fails to mount rw. Message in the boot journal: kernel: F2FS-fs (nvme0n1p6): invalid zstd compress level: 6 There was recently an f2fs patch to the 6.1 kernel tree which might be related: https://www.spinics.net/lists/stable-commits/msg329957.html Was able to recover the system by doing: sudo mount -o remount,rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,compress_algorithm=lz4,compress_log_size=2,compress_mode=fs,atgc,discard_unit=block,memory=normal /dev/nvme0n1p6 / under the running bad 6.1.0-18-amd64 kernel, then editing /etc/default/grub: GRUB_DEFAULT="Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-17-amd64" and running 'update-grub' and rebooting to boot the 6.1.0-17-amd64 kernel. The issue is easily reproducible by: # dd if=/dev/zero of=test.img count=100 bs=1M # mkfs.f2fs -f -O compression,extra_attr ./test.img # mount -t f2fs -o compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime ./test.img /mnt resulting in [ 60.789982] F2FS-fs (loop0): invalid zstd compress level: 6 Hi Salvatore, Can you please try below fixes: [PATCH 6.1] f2fs: add helper to check compression level https://lore.kernel.org/linux-f2fs-devel/20240212160530.1017205-1-c...@kernel.org [PATCH] f2fs: compress: fix to check zstd compress level correctly in mount option https://lore.kernel.org/linux-f2fs-devel/20240212160818.1020903-1-c...@kernel.org Thanks, A bugzilla report has been submitted in https://bugzilla.kernel.org/show_bug.cgi?id=218471 #regzbot introduced: v6.1.69..v6.1.76 #regzbot link: https://bugs.debian.org/1063422 #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218471 Regards, Salvatore
Bug#955549: [f2fs-dev] Bug#955549: f2fs-tools: fsck.f2fs segfaults
On 2020/4/10 7:32, Adam Borowski wrote: > On Tue, Apr 07, 2020 at 06:22:19PM +0800, Chao Yu wrote: >> I figured out two patches to fix segfault issues, could you please have >> a try: >> >> fsck.f2fs: fix to check validation of i_xattr_nid >> fsck.f2fs: fix to check validation of block address >> >> In addition, I found that fsck main flow failed because it can not load root >> inode based on wrong block address in nat, so I wrote another patch to enable >> fsck to lookup root inode by traversing all nodes in f2fs main area, and >> relink >> nat to root inode correctly. >> >> fsck.f2fs: lookup and relink root inode > > I still get a segfault: Oops.. > > Program received signal SIGSEGV, Segmentation fault. > 0x5556 in print_inode_info (sbi=0x55584ca0 , > node=0x5558f170, name=) at mount.c:240 > 240 block_t blkaddr = le32_to_cpu(inode->i_addr[i + ofs]); > (gdb) bt > #0 0x5556 in print_inode_info (sbi=0x55584ca0 , > node=0x5558f170, name=) at mount.c:240 > #1 0x55564c4e in print_node_info (sbi=, > node_block=, verbose=) at mount.c:278 > #2 0x5556317f in dump_node (sbi=sbi@entry=0x55584ca0 , > nid=nid@entry=2861, force=force@entry=1) at dump.c:511 > #3 0x55561060 in fsck_verify (sbi=0x55584ca0 ) at > fsck.c:3259 > #4 0x799a in do_fsck (sbi=0x55584ca0 ) at main.c:698 > #5 main (argc=, argv=) at main.c:864 Fixed with [PATCH] fsck.f2fs: fix to avoid overflow during print_inode_info() Thanks, > >> With this patch, image can be fixed and mounted later, although, most of >> files >> were deleted due to seriously damaged f2fs metadata > > Yeah, I've later tested the hardware -- writes to it are borked, so no > complaint against the filesystem failing. I got backups. :) > >> The patches were made on top of dev-test branch of Jaegeuk's tree: >> https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev-test > >>>>>> #0 0x93ec in memcpy (__len=18446744073323892736, >>>>>> __src=0x5560760c, __dest=0x7fffe000) at >>>>>> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 >>> >>> At a glance, immediate reason of this issue is we didn't check >>> inode.i_namelen's >>> validation. >>> >>>>>> #1 convert_encrypted_name (name=name@entry=0x5560760c " ", >>>>>> len=-385658880, new=new@entry=0x7fffe000 " ", enc_name=>>>>> out>) at fsck.c:1132 >>>>>> #2 0x55562286 in print_inode_info (sbi=0x5557db20 , >>>>>> node=0x556075b0, name=1) at mount.c:183 >>>>>> #3 0x55562a46 in print_node_info (sbi=, >>>>>> node_block=, verbose=) at mount.c:277 >>>>>> #4 0x55560d3f in dump_node (sbi=sbi@entry=0x5557db20 >>>>>> , nid=nid@entry=24274, force=force@entry=1) at dump.c:520 >>>>>> #5 0xe94c in fsck_verify (sbi=0x5557db20 ) at >>>>>> fsck.c:2568 >>>>>> #6 0x699b in do_fsck (sbi=0x5557db20 ) at >>>>>> main.c:569 > > > Meow! >
Bug#955549: [f2fs-dev] Bug#955549: f2fs-tools: fsck.f2fs segfaults
Hi Adam, I figured out two patches to fix segfault issues, could you please have a try: fsck.f2fs: fix to check validation of i_xattr_nid fsck.f2fs: fix to check validation of block address In addition, I found that fsck main flow failed because it can not load root inode based on wrong block address in nat, so I wrote another patch to enable fsck to lookup root inode by traversing all nodes in f2fs main area, and relink nat to root inode correctly. fsck.f2fs: lookup and relink root inode With this patch, image can be fixed and mounted later, although, most of files were deleted due to seriously damaged f2fs metadata The patches were made on top of dev-test branch of Jaegeuk's tree: https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev-test Let me know if you have problem with above patches. Thanks, On 2020/4/3 14:37, Chao Yu wrote: > Thanks for forwarding, Ted. > > On 2020/4/3 10:45, Adam Borowski wrote: >> On Thu, Apr 02, 2020 at 03:16:58PM -0400, Theodore Y. Ts'o wrote: >>> On Thu, Apr 02, 2020 at 02:01:26PM +0200, Adam Borowski wrote: >>>> >>>> After a lot of output on a damaged filesystem (SD card copied to an image) >>>> fsck.f2fs dies with: >>>> >>>> - File name : mkfs.ext3.dpkg-new >>>> - File size : 6 (bytes) >>>> >>>> Program received signal SIGSEGV, Segmentation fault. >>>> 0x93ec in memcpy (__len=18446744073323892736, >>>> __src=0x5560760c, __dest=0x7fffe000) at >>>> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 >>>> 34 return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest)); >> >>>> #0 0x93ec in memcpy (__len=18446744073323892736, >>>> __src=0x5560760c, __dest=0x7fffe000) at >>>> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 > > At a glance, immediate reason of this issue is we didn't check > inode.i_namelen's > validation. > >>>> #1 convert_encrypted_name (name=name@entry=0x5560760c " ", >>>> len=-385658880, new=new@entry=0x7fffe000 " ", enc_name=>>> out>) at fsck.c:1132 >>>> #2 0x55562286 in print_inode_info (sbi=0x5557db20 , >>>> node=0x556075b0, name=1) at mount.c:183 >>>> #3 0x55562a46 in print_node_info (sbi=, >>>> node_block=, verbose=) at mount.c:277 >>>> #4 0x55560d3f in dump_node (sbi=sbi@entry=0x5557db20 , >>>> nid=nid@entry=24274, force=force@entry=1) at dump.c:520 >>>> #5 0xe94c in fsck_verify (sbi=0x5557db20 ) at >>>> fsck.c:2568 >>>> #6 0x699b in do_fsck (sbi=0x5557db20 ) at >>>> main.c:569 >> >>>> I have a copy of the filesystem in question from before any repair >>>> attempts. >>>> It has no sensitive data on it, thus I can share if needed -- 14GB. >>> >>> Thanks for the bug report. Can you make the file system image >>> available somehow? Maybe for download at some URL? How well does it >>> compress? >> >> 916MB -- https://angband.pl/rigel.f2fs.xz.gpg >> The machine serves as a serial console logger/management for a bunch of >> boxes; a root session is unlikely to have anything I'd not share with >> developers but is not something to release to the wide world. Thus, I >> symetrically encrypted the image, I'll send you the password privately -- >> feel free to share it with anyone semi-trusted. > > Would you mind sharing the password with me (c...@kernel.org)? I guess > I can take a look at this issue. > > Thanks, > >> >> The filesystem was on a SD card, with very light use (append), no issues, >> ran kernel 4.13 until 9 days ago -- then I've upgraded to 5.5.11 with no >> other changes. The corruption is probably caused by that, but there's >> always a chance of SD being SD. >> >> >> Meow! >> > > > ___ > Linux-f2fs-devel mailing list > linux-f2fs-de...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel > . >
Bug#955549: [f2fs-dev] Bug#955549: f2fs-tools: fsck.f2fs segfaults
Thanks for forwarding, Ted. On 2020/4/3 10:45, Adam Borowski wrote: > On Thu, Apr 02, 2020 at 03:16:58PM -0400, Theodore Y. Ts'o wrote: >> On Thu, Apr 02, 2020 at 02:01:26PM +0200, Adam Borowski wrote: >>> >>> After a lot of output on a damaged filesystem (SD card copied to an image) >>> fsck.f2fs dies with: >>> >>> - File name : mkfs.ext3.dpkg-new >>> - File size : 6 (bytes) >>> >>> Program received signal SIGSEGV, Segmentation fault. >>> 0x93ec in memcpy (__len=18446744073323892736, >>> __src=0x5560760c, __dest=0x7fffe000) at >>> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 >>> 34return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest)); > >>> #0 0x93ec in memcpy (__len=18446744073323892736, >>> __src=0x5560760c, __dest=0x7fffe000) at >>> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 At a glance, immediate reason of this issue is we didn't check inode.i_namelen's validation. >>> #1 convert_encrypted_name (name=name@entry=0x5560760c " ", >>> len=-385658880, new=new@entry=0x7fffe000 " ", enc_name=) >>> at fsck.c:1132 >>> #2 0x55562286 in print_inode_info (sbi=0x5557db20 , >>> node=0x556075b0, name=1) at mount.c:183 >>> #3 0x55562a46 in print_node_info (sbi=, >>> node_block=, verbose=) at mount.c:277 >>> #4 0x55560d3f in dump_node (sbi=sbi@entry=0x5557db20 , >>> nid=nid@entry=24274, force=force@entry=1) at dump.c:520 >>> #5 0xe94c in fsck_verify (sbi=0x5557db20 ) at >>> fsck.c:2568 >>> #6 0x699b in do_fsck (sbi=0x5557db20 ) at main.c:569 > >>> I have a copy of the filesystem in question from before any repair >>> attempts. >>> It has no sensitive data on it, thus I can share if needed -- 14GB. >> >> Thanks for the bug report. Can you make the file system image >> available somehow? Maybe for download at some URL? How well does it >> compress? > > 916MB -- https://angband.pl/rigel.f2fs.xz.gpg > The machine serves as a serial console logger/management for a bunch of > boxes; a root session is unlikely to have anything I'd not share with > developers but is not something to release to the wide world. Thus, I > symetrically encrypted the image, I'll send you the password privately -- > feel free to share it with anyone semi-trusted. Would you mind sharing the password with me (c...@kernel.org)? I guess I can take a look at this issue. Thanks, > > The filesystem was on a SD card, with very light use (append), no issues, > ran kernel 4.13 until 9 days ago -- then I've upgraded to 5.5.11 with no > other changes. The corruption is probably caused by that, but there's > always a chance of SD being SD. > > > Meow! >
