Kernel release bisection: cosmic: 4.18.0-26: incremental, dry-run: panic about 303GB cosmic: 4.18.0-26: incremental, dry-run: panic about 21GB cosmic: 4.18.0-26: incremental, dry-run: panic about 22GB
disco: 4.19.0-13: incremental, dry-run: testing: success The first run using 4.18.0-26 that was able to get to 303GB before panic is intriguing (it's why I ran it twice). All previous panics when doing an incremental rsync were ~20-22GB in (ala the subsequent two runs). The only thing we can say with certainty is the problem did not reproduce using 4.19.0-13. If the problem is a stack overflow then any changes in stack size, or different paths which use less stack, would result in the problem being less visible using the amount of data I'm copying (~1.4TB). ** Attachment added: "crash files from weekend of 10/3/20" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1895563/+attachment/5417903/+files/crash_201003.tar.bz2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1895563 Title: Kernel Oops: Rsyncing to bcache device w/o backing cache kernel panic Status in linux package in Ubuntu: Confirmed Bug description: Platform details: 1) version.log attached 2) lspci-vnvn.log attached 3) lsb_release -rd Description: Ubuntu 18.04.5 LTS Release: 18.04 4) apt-cache policy bcache-tools bcache-tools: Installed: 1.0.8-2ubuntu0.18.04.1 Candidate: 1.0.8-2ubuntu0.18.04.1 Version table: *** 1.0.8-2ubuntu0.18.04.1 500 500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 100 /var/lib/dpkg/status 1.0.8-2build1 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages 5) uname -a: Linux timber4 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Short problem description: Rsyncing from a 1.4TB home dir to a an new filesystem on a bcache device without an attached cache eventually results in kernel panic. What I expected to happen: no kernel panic. Long problem description: 1) Devices / Filesystems: bfd01: 8*4TB configured as RAID6 using md LVM2 volume group. bfd02: 8*4TB configured as RAID6 using Dell PERC H810 LVM2 volume group. The tree contains 1069685 files / dirs. 2) Steps to reproduce: # Make the device, format and mount it make-bcache -B /dev/vg-bfd02/delme01_bc mkfs.xfs -f -L delme01bc /dev/bcache0 mount /dev/bcache0 /dev/bfd02/delme01bc # Rsync flags="-a --delete" ; info_cmd="--info=progress2" ; excludes=""; src=/mnt/bfd01/delme01 ; tgt=/mnt/bfd02/delme01bc/ ; time rsync $flags $info_cmd $excludes $src/ $tgt/ ... # eventually kernel panic Using incremental rsync it will kernel panic after reading about 21GB. # If instead I use non-incremental rsync it will kernel panic after reading about 1.2-1.3TB. flags="--no-inc-recursive -ax -HAXS --delete" ; info_cmd="--info=progress2" ; excludes=""; ... # eventually kernel panic Additional detail: 1) I was *not* able to reproduce if bfd01/delm01bc was a target and bfd02 was the source e.g. # Rsync flags="-a --delete" ; info_cmd="--info=progress2" ; excludes=""; src=/mnt/bfd02/delme01 ; tgt=/mnt/bfd01/delme01bc/ ; time rsync $flags $info_cmd $excludes $src/ $tgt/ ... # rsync completes successfully 2) I was *not* able to reproduce if I attached and then detached a cache set: ceho 'd3b93488-714e-4efa-af94-cd80fd2db11f' > /sys/block/bcache0/bcache/attach echo 1 > /sys/block/bcache0/bcache/detach ... time rsync $flags $info_cmd $excludes $src/ $tgt/ ... # rsync completes successfully I have 11 compressed kernel crash dumps. I will upload 3 after filing this. Let me know if you want the other 8. ProblemType: KernelCrash DistroRelease: Ubuntu 18.04 Package: linux-image-4.15.0-117-generic 4.15.0-117.118 ProcVersionSignature: Ubuntu 4.15.0-117.118-generic 4.15.18 Uname: Linux 4.15.0-117-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Sep 13 12:53 seq crw-rw---- 1 root audio 116, 33 Sep 13 12:53 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.17 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Sat Sep 12 09:51:37 2020 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: Dell Inc. PowerEdge R620 PciMultimedia: ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-117-generic root=UUID=de68d15b-3948-4666-9bc5-5cbecc83971c ro maybe-ubiquity crashkernel=512M-:512M RelatedPackageVersions: linux-restricted-modules-4.15.0-117-generic N/A linux-backports-modules-4.15.0-117-generic N/A linux-firmware 1.173.19 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/06/2019 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.9.0 dmi.board.name: 0KCKR5 dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.9.0:bd12/06/2019:svnDellInc.:pnPowerEdgeR620:pvr:rvnDellInc.:rn0KCKR5:rvrA00:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R620 dmi.sys.vendor: Dell Inc. --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Sep 13 12:53 seq crw-rw---- 1 root audio 116, 33 Sep 13 12:53 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.17 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 18.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: Dell Inc. PowerEdge R620 Package: linux (not installed) PciMultimedia: ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-117-generic root=UUID=de68d15b-3948-4666-9bc5-5cbecc83971c ro maybe-ubiquity crashkernel=512M-:512M ProcVersionSignature: Ubuntu 4.15.0-117.118-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-117-generic N/A linux-backports-modules-4.15.0-117-generic N/A linux-firmware 1.173.19 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic uec-images Uname: Linux 4.15.0-117-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/06/2019 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.9.0 dmi.board.name: 0KCKR5 dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.9.0:bd12/06/2019:svnDellInc.:pnPowerEdgeR620:pvr:rvnDellInc.:rn0KCKR5:rvrA00:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R620 dmi.sys.vendor: Dell Inc. --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Sep 13 12:53 seq crw-rw---- 1 root audio 116, 33 Sep 13 12:53 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.17 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 18.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: Dell Inc. PowerEdge R620 Package: linux (not installed) PciMultimedia: ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-117-generic root=UUID=de68d15b-3948-4666-9bc5-5cbecc83971c ro maybe-ubiquity crashkernel=512M-:512M ProcVersionSignature: Ubuntu 4.15.0-117.118-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-117-generic N/A linux-backports-modules-4.15.0-117-generic N/A linux-firmware 1.173.19 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic uec-images Uname: Linux 4.15.0-117-generic x86_64 UnreportableReason: This report is about a package that is not installed. UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: False dmi.bios.date: 12/06/2019 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.9.0 dmi.board.name: 0KCKR5 dmi.board.vendor: Dell Inc. dmi.board.version: A00 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.9.0:bd12/06/2019:svnDellInc.:pnPowerEdgeR620:pvr:rvnDellInc.:rn0KCKR5:rvrA00:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R620 dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1895563/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp