Thanks for fixing this issue. We confirm that the problem is gone with updated kernel.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823972 Title: bionic, xenial/hwe: misses "fuse: fix initial parallel dirops" patch Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: Fix Released Bug description: SRU Justification: [Impact] * Enabling parallel dirops in fuse (FUSE_PARALLEL_DIROPS) may cause a race condition that leave fuse inode's mutex held, triggering a deadlock * The problem is that the lock and unlock paths are relying on get_fuse_conn(inode)->parallel_dirops to decide if the mutex needs to be acquired/released, but its value might be set in the lock path and unset in the unlock path (leaving the mutex held) [Test Case] * A test case that triggers the bug almost immediately can be found here https://github.com/hanwen/go-fuse/pull/288 [Fix] * Instead of relying on get_fuse_conn(inode)->parallel_dirops both in fuse_lock_inode() and fuse_unlock_inode(), only check this flag in the locking path and pass a variable to fuse_unlock_inode() to determine if the mutex was acquired or not [Regression Potential] * Fix has been tested on the affected platform. It is an upstream fix that seems to affect only 4.7+ kernels, more exactly in our case only Bionic kernels (and derived) are affected. Cosmic and above already include this fix. So regression potential is minimal. [Original bug report] Hello up there, We were reported about a deadlock in the kernel while using a FUSE-based filesystem on Ubuntu. The kernel in question is Ubuntu-hwe-4.15.0-47.50~16.04.1 from Xenial/HWE. We tracked this bug to the fact that 4.15.x kernel in Ubuntu does not include the following patch, in despite the patch being marked as needed for v4.7+ stable kernels: https://git.kernel.org/linus/63576c13bd Please see the following go-fuse issue for full details: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480008562 The bug is potentially applicable to libfuse users too since libfuse by default enables parallel dirops whenever kernel claims support for it, which libfuse maintained confirmed: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480013202 https://github.com/hanwen/go-fuse/issues/281#issuecomment-480510381 We tested that cherry-picking 63576c13bd into 4.15.x series makes the problem go away: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480499969 So please include https://git.kernel.org/linus/63576c13bd into Ubuntu 4.15.x kernel series which are bionic/master and xenial/hwe, and which currently don't have this patch. -------- Here is a full list of FUSE patches marked to be needed in stable kernels starting from v4.15: kirr@deco:~/src/linux/linux$ git log --oneline v4.15..v5.1-rc3 --grep="stable@" -- fs/fuse/ a2ebba824106 fuse: decrement NR_WRITEBACK_TEMP on the right page 9509941e9c53 fuse: call pipe_buf_release() under pipe lock 8a3177db59cd cuse: fix ioctl 97e1532ef81a fuse: handle zero sized retrieve correctly 2e64ff154ce6 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS ebacb8127359 fuse: fix use-after-free in fuse_direct_IO() 2d84a2d19b61 fuse: fix possibly missed wake-up after abort 7fabaf303458 fuse: fix leaked notify reply 908a572b80f6 fuse: fix blocked_waitq wakeup 4c316f2f3ff3 fuse: set FR_SENT while locked d2d2d4fb1f54 fuse: Fix use-after-free in fuse_dev_do_write() bc78abbd55dd fuse: Fix use-after-free in fuse_dev_do_read() a2477b0e67c5 fuse: Don't access pipe->buffers without pipe_lock() 63576c13bd17 fuse: fix initial parallel dirops e8f3bd773d22 fuse: Fix oops at process_init_reply() b8f95e5d13f5 fuse: umount should wait for all requests 45ff350bbd9d fuse: fix unlocked access to processing queue 87114373ea50 fuse: fix double request_end() 543b8f8662fe (tag: fuse-update-4.18) fuse: don't keep dead fuse_conn at fuse_fill_super(). 6becdb601bae fuse: fix control dir setup and teardown 8a301eb16d99 fuse: fix congested state leak on aborted connections df0e91d48827 fuse: atomic_o_trunc should truncate pagecache Among those only 8a3177db59cd and 2d84a2d19b61 should not be applied to 4.15.x becuase they cure a problem introduced in a later kernel (please see got log without --oneline for stable@ details) However both bionic and xenial/hwe has much less fuse patches applied: kirr@deco:~/src/linux/linux$ git log --oneline v4.15..bionic/master -- fs/fuse/ e992e3521885 fuse: fix control dir setup and teardown f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super(). 840c77082f93 fuse: atomic_o_trunc should truncate pagecache c0e31b214498 fuse: fix congested state leak on aborted connections 45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts 1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns kirr@deco:~/src/linux/linux$ git log --oneline v4.15..xenial/hwe -- fs/fuse/ e992e3521885 fuse: fix control dir setup and teardown f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super(). 840c77082f93 fuse: atomic_o_trunc should truncate pagecache c0e31b214498 fuse: fix congested state leak on aborted connections 45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts 1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns which suggests that other FUSE fixes should be cherry-picked too. Please consider cherry-picking those additional patches too. They are all in upstream kernel stable series, e.g. stable/linux-4.14.y has them: kirr@deco:~/src/linux/linux$ git log --oneline v4.14..stable/linux-4.14.y -- fs/fuse/ 266a69895b89 fuse: handle zero sized retrieve correctly b928e93d864c fuse: decrement NR_WRITEBACK_TEMP on the right page 65f222bb370e fuse: call pipe_buf_release() under pipe lock c1149b873482 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS 6ceec07cc84a fuse: fix leaked notify reply a42d933dc281 fuse: fix use-after-free in fuse_direct_IO() 78da72ee42d8 fuse: set FR_SENT while locked f6f21a2b70c6 fuse: fix blocked_waitq wakeup ab962e91008a fuse: Fix use-after-free in fuse_dev_do_write() d94b3a2375cb fuse: Fix use-after-free in fuse_dev_do_read() e8a3f3a03655 fuse: Add missed unlock_page() to fuse_readpages_fill() ff4a71855d0a fuse: Fix oops at process_init_reply() 973206923812 fuse: umount should wait for all requests fc17d7519e8e fuse: fix unlocked access to processing queue cfb6eca6e4bb fuse: fix double request_end() 7d392674443c fuse: fix initial parallel dirops eaebcf902ae0 fuse: Don't access pipe->buffers without pipe_lock() 69829f749a43 fuse: fix control dir setup and teardown 3a37d85a90da fuse: don't keep dead fuse_conn at fuse_fill_super(). 2f7bf369b5f8 fuse: atomic_o_trunc should truncate pagecache 02832578eb9d fuse: fix congested state leak on aborted connections and it is just that stable/linux-4.15.y stopped being maintained by Greg KH. Thanks beforehand, Kirill --- ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CurrentDesktop: Unity DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=dec25862-42c7-4494-8f75-e6cc76aa65ea InstallationDate: Installed on 2019-02-28 (39 days ago) InstallationMedia: Ubuntu 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190227) IwConfig: lo no wireless extensions. ens3 no wireless extensions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) Package: linux (not installed) ProcFB: 0 virtiodrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic root=UUID=fe1f50d9-7142-4ee3-8904-bae7d80bc6b1 ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-45-generic N/A linux-backports-modules-4.15.0-45-generic N/A linux-firmware 1.157.21 RfKill: Tags: xenial Uname: Linux 4.15.0-45-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo _MarkForUpload: True dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.12.0-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-3.1 dmi.modalias: dmi:bvnSeaBIOS:bvr1.12.0-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-3.1:cvnQEMU:ct1:cvrpc-i440fx-3.1: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-3.1 dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1823972/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp