[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
For posterity: If https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/comments/143 is the cause of this issue for you, dmesg -D (which turns off console logging of kernel messages) might be a viable workaround until you can reboot. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug was fixed in the package linux - 3.13.0-83.127 --- linux (3.13.0-83.127) trusty; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555839 [ Florian Westphal ] * SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving userspace - LP: #1555338 linux (3.13.0-82.126) trusty; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1554732 [ Upstream Kernel Changes ] * Revert "drm/radeon: call hpd_irq_event on resume" - LP: #1554608 * net: generic dev_disable_lro() stacked device handling - LP: #1547680 linux (3.13.0-81.125) trusty; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1552316 [ Upstream Kernel Changes ] * Revert "firmware: dmi_scan: Fix UUID endianness for SMBIOS >= 2.6" - LP: #1551419 * bcache: Fix a lockdep splat in an error path - LP: #1551327 linux (3.13.0-80.124) trusty; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1548519 [ Andy Whitcroft ] * [Debian] hv: hv_set_ifconfig -- convert to python3 - LP: #1506521 * [Debian] hv: hv_set_ifconfig -- switch to approved indentation - LP: #1540586 * [Debian] hv: hv_set_ifconfig -- fix numerous parameter handling issues - LP: #1540586 [ Dan Streetman ] * SAUCE: nbd: ratelimit error msgs after socket close - LP: #1505564 [ Upstream Kernel Changes ] * Revert "workqueue: make sure delayed work run in local cpu" - LP: #1546320 * [media] gspca: ov534/topro: prevent a division by 0 - LP: #1542497 * [media] media: dvb-core: Don't force CAN_INVERSION_AUTO in oneshot mode - LP: #1542497 * tools lib traceevent: Fix output of %llu for 64 bit values read on 32 bit machines - LP: #1542497 * KVM: x86: correctly print #AC in traces - LP: #1542497 * drm/radeon: call hpd_irq_event on resume - LP: #1542497 * xhci: refuse loading if nousb is used - LP: #1542497 * arm64: Clear out any singlestep state on a ptrace detach operation - LP: #1542497 * time: Avoid signed overflow in timekeeping_get_ns() - LP: #1542497 * rtlwifi: fix memory leak for USB device - LP: #1542497 * wlcore/wl12xx: spi: fix oops on firmware load - LP: #1542497 * EDAC, mc_sysfs: Fix freeing bus' name - LP: #1542497 * EDAC: Don't try to cancel workqueue when it's never setup - LP: #1542497 * EDAC: Robustify workqueues destruction - LP: #1542497 * powerpc: Make value-returning atomics fully ordered - LP: #1542497 * powerpc: Make {cmp}xchg* and their atomic_ versions fully ordered - LP: #1542497 * dm space map metadata: remove unused variable in brb_pop() - LP: #1542497 * dm thin: fix race condition when destroying thin pool workqueue - LP: #1542497 * futex: Drop refcount if requeue_pi() acquired the rtmutex - LP: #1542497 * drm/radeon: clean up fujitsu quirks - LP: #1542497 * mmc: sdio: Fix invalid vdd in voltage switch power cycle - LP: #1542497 * mmc: sdhci: Fix sdhci_runtime_pm_bus_on/off() - LP: #1542497 * udf: limit the maximum number of indirect extents in a row - LP: #1542497 * nfs: Fix race in __update_open_stateid() - LP: #1542497 * USB: cp210x: add ID for ELV Marble Sound Board 1 - LP: #1542497 * NFSv4: Don't perform cached access checks before we've OPENed the file - LP: #1542497 * NFS: Fix attribute cache revalidation - LP: #1542497 * posix-clock: Fix return code on the poll method's error path - LP: #1542497 * rtlwifi: rtl8192de: Fix incorrect module parameter descriptions - LP: #1542497 * rtlwifi: rtl8192se: Fix module parameter initialization - LP: #1542497 * rtlwifi: rtl8192ce: Fix handling of module parameters - LP: #1542497 * rtlwifi: rtl8192cu: Add missing parameter setup - LP: #1542497 * bcache: fix a livelock when we cause a huge number of cache misses - LP: #1542497 * bcache: Add a cond_resched() call to gc - LP: #1542497 * bcache: clear BCACHE_DEV_UNLINK_DONE flag when attaching a backing device - LP: #1542497 * bcache: fix a leak in bch_cached_dev_run() - LP: #1542497 * bcache: unregister reboot notifier if bcache fails to unregister device - LP: #1542497 * bcache: add mutex lock for bch_is_open - LP: #1542497 * bcache: allows use of register in udev to avoid "device_busy" error. - LP: #1542497 * bcache: Change refill_dirty() to always scan entire disk if necessary - LP: #1542497 * wlcore/wl12xx: spi: fix NULL pointer dereference (Oops) - LP: #1542497 * Input: i8042 - add Fujitsu Lifebook U745 to the nomux list - LP: #1542497 * libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct - LP: #1542497 * x86/xen: don't reset vcpu_info on a cancelled suspend - LP: #1542497 * udf: Prevent buffer overrun with multi-byte characters - LP: #1542497 * udf: Check output buffer length when converting name to CS0 - LP: #1542497
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug was fixed in the package linux - 3.19.0-56.62 --- linux (3.19.0-56.62) vivid; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555832 [ Florian Westphal ] * SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving userspace - LP: #1555338 linux (3.19.0-55.61) vivid; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1554708 [ Upstream Kernel Changes ] * Revert "drm/radeon: call hpd_irq_event on resume" - LP: #1554608 linux (3.19.0-54.60) vivid; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1552337 [ Upstream Kernel Changes ] * Revert "firmware: dmi_scan: Fix UUID endianness for SMBIOS >= 2.6" - LP: #1551419 linux (3.19.0-53.59) vivid; urgency=low [ Kamal Mostafa ] * Release Tracking Bug - LP: #1550576 [ Kamal Mostafa ] * Merged back 3.19.0-52.58 linux (3.19.0-52.58) vivid; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1548548 [ Dan Streetman ] * SAUCE: nbd: ratelimit error msgs after socket close - LP: #1505564 [ Upstream Kernel Changes ] * Revert "ACPI / LPSS: allow to use specific PM domain during ->probe()" - LP: #1542457 * Revert "workqueue: make sure delayed work run in local cpu" - LP: #1546320 * net: ipmr: fix static mfc/dev leaks on table destruction - LP: #1542457 * drm/nouveau/nv46: Change mc subdev oclass from nv44 to nv4c - LP: #1542457 * ovl: allow zero size xattr - LP: #1542457 * ovl: use a minimal buffer in ovl_copy_xattr - LP: #1542457 * [media] vb2: fix a regression in poll() behavior for output,streams - LP: #1542457 * [media] gspca: ov534/topro: prevent a division by 0 - LP: #1542457 * [media] media: dvb-core: Don't force CAN_INVERSION_AUTO in oneshot mode - LP: #1542457 * tools lib traceevent: Fix output of %llu for 64 bit values read on 32 bit machines - LP: #1542457 * KVM: x86: expose MSR_TSC_AUX to userspace - LP: #1542457 * KVM: x86: correctly print #AC in traces - LP: #1542457 * drm/radeon: call hpd_irq_event on resume - LP: #1542457 * xhci: refuse loading if nousb is used - LP: #1542457 * arm64: Clear out any singlestep state on a ptrace detach operation - LP: #1542457 * time: Avoid signed overflow in timekeeping_get_ns() - LP: #1542457 * ovl: root: copy attr - LP: #1542457 * Bluetooth: Add support of Toshiba Broadcom based devices - LP: #1522949, #1542457 * rtlwifi: fix memory leak for USB device - LP: #1542457 * wlcore/wl12xx: spi: fix oops on firmware load - LP: #1542457 * ovl: check dentry positiveness in ovl_cleanup_whiteouts() - LP: #1542457 * EDAC, mc_sysfs: Fix freeing bus' name - LP: #1542457 * EDAC: Robustify workqueues destruction - LP: #1542457 * arm64: mm: ensure that the zero page is visible to the page table walker - LP: #1542457 * powerpc: Make value-returning atomics fully ordered - LP: #1542457 * powerpc: Make {cmp}xchg* and their atomic_ versions fully ordered - LP: #1542457 * dm space map metadata: remove unused variable in brb_pop() - LP: #1542457 * dm thin: fix race condition when destroying thin pool workqueue - LP: #1542457 * futex: Drop refcount if requeue_pi() acquired the rtmutex - LP: #1542457 * arm64: mdscr_el1: avoid exposing DCC to userspace - LP: #1542457 * arm64: kernel: enforce pmuserenr_el0 initialization and restore - LP: #1542457 * drm/radeon: Fix off-by-one errors in radeon_vm_bo_set_addr - LP: #1542457 * drm/radeon: clean up fujitsu quirks - LP: #1542457 * mmc: sdio: Fix invalid vdd in voltage switch power cycle - LP: #1542457 * mmc: sdhci: Fix DMA descriptor with zero data length - LP: #1542457 * mmc: sdhci: Fix sdhci_runtime_pm_bus_on/off() - LP: #1542457 * udf: limit the maximum number of indirect extents in a row - LP: #1542457 * [media] rc: sunxi-cir: Initialize the spinlock properly - LP: #1542457 * nfs: Fix race in __update_open_stateid() - LP: #1542457 * USB: cp210x: add ID for ELV Marble Sound Board 1 - LP: #1542457 * NFSv4: Don't perform cached access checks before we've OPENed the file - LP: #1542457 * NFS: Ensure we revalidate attributes before using execute_ok() - LP: #1542457 * Thermal: initialize thermal zone device correctly - LP: #1542457 * Thermal: handle thermal zone device properly during system sleep - LP: #1542457 * Thermal: do thermal zone update after a cooling device registered - LP: #1542457 * posix-clock: Fix return code on the poll method's error path - LP: #1542457 * rtlwifi: rtl8723be: Fix module parameter initialization - LP: #1542457 * rtlwifi: rtl8723ae: Fix initialization of module parameters - LP: #1542457 * rtlwifi: rtl8821ae: Fix errors in parameter initialization - LP: #1542457 * rtlwifi: rtl8188ee: Fix module parameter initialization
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug was fixed in the package linux - 4.2.0-34.39 --- linux (4.2.0-34.39) wily; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555821 [ Florian Westphal ] * SAUCE: [nf] netfilter: x_tables: check for size overflow - LP: #1555353 * SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving userspace - LP: #1555338 linux (4.2.0-33.38) wily; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1554649 [ Upstream Kernel Changes ] * Revert "drm/radeon: call hpd_irq_event on resume" - LP: #1554608 * cxl: Fix PSL timebase synchronization detection - LP: #1532914 linux (4.2.0-32.37) wily; urgency=low [ Kamal Mostafa ] * Release Tracking Bug - LP: #1550045 [ Kamal Mostafa ] * Merged back Ubuntu-4.2.0-31.36 linux (4.2.0-31.36) wily; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1548579 [ Andy Whitcroft ] * [Debian] hv: hv_set_ifconfig -- convert to python3 - LP: #1506521 * [Debian] hv: hv_set_ifconfig -- switch to approved indentation - LP: #1540586 * [Debian] hv: hv_set_ifconfig -- fix numerous parameter handling issues - LP: #1540586 [ Carol L Soto ] * SAUCE: IB/IPoIB: Do not set skb truesize since using one linearskb - LP: #1541326 [ Dan Streetman ] * SAUCE: nbd: ratelimit error msgs after socket close - LP: #1505564 [ Tim Gardner ] * Revert "SAUCE: (noup) cxlflash: Fix to avoid virtual LUN failover failure" - LP: #1541635 * Revert "SAUCE: (noup) cxlflash: Fix to escalate LINK_RESET also on port 1" - LP: #1541635 * [Config] ARMV8_DEPRECATED=y - LP: #1545542 [ Upstream Kernel Changes ] * x86/xen/p2m: hint at the last populated P2M entry - LP: #1542941 * mm: add dma_pool_zalloc() call to DMA API - LP: #1543737 * sctp: Prevent soft lockup when sctp_accept() is called during a timeout event - LP: #1543737 * xen-netback: respect user provided max_queues - LP: #1543737 * xen-netfront: respect user provided max_queues - LP: #1543737 * xen-netfront: update num_queues to real created - LP: #1543737 * iio: adis_buffer: Fix out-of-bounds memory access - LP: #1543737 * KVM: PPC: Fix emulation of H_SET_DABR/X on POWER8 - LP: #1543737 * KVM: PPC: Fix ONE_REG AltiVec support - LP: #1543737 * x86/irq: Call chip->irq_set_affinity in proper context - LP: #1543737 * drm/amdgpu: fix tonga smu resume - LP: #1543737 * perf kvm record/report: 'unprocessable sample' error while recording/reporting guest data - LP: #1543737 * hrtimer: Handle remaining time proper for TIME_LOW_RES - LP: #1543737 * timerfd: Handle relative timers with CONFIG_TIME_LOW_RES proper - LP: #1543737 * posix-timers: Handle relative timers with CONFIG_TIME_LOW_RES proper - LP: #1543737 * itimers: Handle relative timers with CONFIG_TIME_LOW_RES proper - LP: #1543737 * drm/amdgpu: Use drm_calloc_large for VM page_tables array - LP: #1543737 * drm/amdgpu: fix amdgpu_bo_pin_restricted VRAM placing v2 - LP: #1543737 * drm/radeon: properly byte swap vce firmware setup - LP: #1543737 * ACPI: Revert "ACPI / video: Add Dell Inspiron 5737 to the blacklist" - LP: #1543737 * ACPI / PCI / hotplug: unlock in error path in acpiphp_enable_slot() - LP: #1543737 * hwmon: (dell-smm) Blacklist Dell Studio XPS 8000 - LP: #1543737 * usb: cdc-acm: handle unlinked urb in acm read callback - LP: #1543737 * usb: cdc-acm: send zero packet for intel 7260 modem - LP: #1543737 * cdc-acm:exclude Samsung phone 04e8:685d - LP: #1543737 * usb: hub: do not clear BOS field during reset device - LP: #1543737 * USB: cp210x: add ID for IAI USB to RS485 adaptor - LP: #1543737 * USB: visor: fix null-deref at probe - LP: #1543737 * USB: serial: visor: fix crash on detecting device without write_urbs - LP: #1543737 * USB: serial: option: Adding support for Telit LE922 - LP: #1543737 * ALSA: seq: Fix incorrect sanity check at snd_seq_oss_synth_cleanup() - LP: #1543737 * ALSA: seq: Degrade the error message for too many opens - LP: #1543737 * USB: serial: ftdi_sio: add support for Yaesu SCU-18 cable - LP: #1543737 * arm64: kernel: fix architected PMU registers unconditional access - LP: #1543737 * USB: option: fix Cinterion AHxx enumeration - LP: #1543737 * ALSA: compress: Disable GET_CODEC_CAPS ioctl for some architectures - LP: #1543737 * ALSA: usb-audio: Fix TEAC UD-501/UD-503/NT-503 usb delay - LP: #1543737 * virtio_pci: fix use after free on release - LP: #1543737 * ALSA: bebob: Use a signed return type for get_formation_index - LP: #1543737 * arm64: errata: Add -mpc-relative-literal-loads to build flags - LP: #1533009, #1543737 * arm64: mm: avoid calling apply_to_page_range on empty range - LP: #1543737 * x86/mm: Fix types used in pgprot cacheability
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Thanks ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
verification can be done with this script: #!/bin/bash modprobe nbd qemu-nbd -d /dev/nbd0 truncate /tmp/testfile -s 20G qemu-nbd -c /dev/nbd0 /tmp/testfile for n in $( seq 1 250 ) ; do echo $n ( dd if=/dev/zero of=/dev/nbd0 bs=1 & ) done qemu-nbd -d /dev/nbd0 after running that, on an unpatched system the dmesg will show a large number (~100 or more) of messages like: [ 70.408246] block nbd0: Attempted send on closed socket with a patched kernel, the dmesg will show a ratelimited number (~10) of those messages. This has been verified on trusty 3.13, vivid 3.19, and wily 4.2 ** Tags removed: verification-needed-vivid verification-needed-wily ** Tags added: verification-done-vivid verification-done-wily -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Tags removed: verification-needed-trusty ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Branch linked: lp:ubuntu/trusty-proposed/linux-lts-vivid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- vivid' to 'verification-done-vivid'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-wily -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- wily' to 'verification-done-wily'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-trusty ** Tags added: verification-needed-vivid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Also affects: linux (Ubuntu Vivid) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Wily) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Vivid) Status: New => Fix Committed ** Changed in: linux (Ubuntu Wily) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Trusty) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This bug was fixed in the package linux - 4.4.0-6.21 --- linux (4.4.0-6.21) xenial; urgency=low [ Tim Gardner ] * Release Tracking Bug - LP: #1546283 * Naples/Zen, NTB Driver (LP: #1542071) - [Config] CONFIG_NTB_AMD=m - NTB: Add support for AMD PCI-Express Non-Transparent Bridge * [Hyper-V] kernel panic occurs when installing Ubuntu Server x32 (LP: #1495983) - SAUCE: storvsc: use small sg_tablesize on x86 * Enable arm64 emulation of removed ARMv7 instructions (LP: #1545542) - [Config] CONFIG_ARMV8_DEPRECATED=y * Surelock-GA2:kernel panic/ exception @ pcibios_set_pcie_reset_state+0x118/0x280 + cxl_reset+0x5c/0xc0 (LP: #1545037) - powerpc/eeh: Fix stale cached primary bus * Miscellaneous Ubuntu changes - SAUCE: fs: Add user namesapace member to struct super_block - SAUCE: fs: Limit file caps to the user namespace of the super block - SAUCE: Smack: Add support for unprivileged mounts from user namespaces - SAUCE: block_dev: Support checking inode permissions in lookup_bdev() - SAUCE: block_dev: Check permissions towards block device inode when mounting - SAUCE: fs: Treat foreign mounts as nosuid - SAUCE: selinux: Add support for unprivileged mounts from user namespaces - SAUCE: userns: Replace in_userns with current_in_userns - SAUCE: Smack: Handle labels consistently in untrusted mounts - SAUCE: fs: Check for invalid i_uid in may_follow_link() - SAUCE: cred: Reject inodes with invalid ids in set_create_file_as() - SAUCE: fs: Refuse uid/gid changes which don't map into s_user_ns - SAUCE: fs: Update posix_acl support to handle user namespace mounts - SAUCE: fs: Ensure the mounter of a filesystem is privileged towards its inodes - SAUCE: fs: Don't remove suid for CAP_FSETID in s_user_ns - SAUCE: fs: Allow superblock owner to access do_remount_sb() - SAUCE: capabilities: Allow privileged user in s_user_ns to set security.* xattrs - SAUCE: fuse: Add support for pid namespaces - SAUCE: fuse: Support fuse filesystems outside of init_user_ns - SAUCE: fuse: Restrict allow_other to the superblock's namespace or a descendant - SAUCE: fuse: Allow user namespace mounts - SAUCE: mtd: Check permissions towards mtd block device inode when mounting - SAUCE: fs: Update i_[ug]id_(read|write) to translate relative to s_user_ns - SAUCE: quota: Convert ids relative to s_user_ns - SAUCE: evm: Translate user/group ids relative to s_user_ns when computing HMAC - SAUCE: fs: Allow CAP_SYS_ADMIN in s_user_ns to freeze and thaw filesystems - SAUCE: quota: Treat superblock owner as privilged - SAUCE: ima/evm: Allow root in s_user_ns to set xattrs - SAUCE: block_dev: Forbid unprivileged mounting when device is opened for writing - SAUCE: ext4: Add support for unprivileged mounts from user namespaces - SAUCE: ext4: Add module parameter to enable user namespace mounts - SAUCE: fuse: Add module parameter to enable user namespace mounts * Miscellaneous upstream changes - megaraid: Fix possible NULL pointer deference in mraid_mm_ioctl - libahci: Implement the capability to override the generic ahci interrupt handler. - ata: Remove the AHCI_HFLAG_EDGE_IRQ support from libahci. - ahci_xgene: Implement the workaround to fix the missing of the edge interrupt for the HOST_IRQ_STAT. -- Tim Gardner Fri, 12 Feb 2016 09:49:05 -0700 ** Changed in: linux (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
opened bug 1534216 to track the serial port issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Tags added: canonical-bootstack -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Great. I'll send the patch upstream, and open a new bug for the serial port hanging issue. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I resumed the tests on Monday, and so far we're looking good. Your change prevented ~10 locks so far, it would seem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I applied the patch, and it saved a reboot twice already, I think. dmesg from one server : http://pastebin.ubuntu.com/14438525/ I have to stop the tests for the weekend though, I'll resume on Monday. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Ok, nm about the sosreport - I got the info from some older emails from axino, nova is using qemu-nbd to locally mount images and access the partitions inside them. I was able to trivially reproduce this simply by creating an image, attaching it with qemu-nbd to /dev/nbd0, partitioning it and mkfs its p1 and then mounting it, then while copying a file to it, performing qemu-nbd -d to un-attach it to /dev/nbd0. That causes the spam of "Attempted..." error messages. So this appears to be a simple case of nova calling qemu-nbd -d while there is still I/O to the image. The right thing to do is simply ratelimit the error messages (and they really should be anyway, as they're printing directly inside a loop). The messages themselves do not indicate any kernel error, simply that the nbd device was removed while being written to. Can you try this kernel PPA to see if it fixes the problem? You will still see the error messages, but only a few lines since they'll be ratelimited. Of course there is still the (probably more serious) problem of the serial port driver hanging a cpu and eating up memory; that probably deserves its own bug, since it's caused by this, but a separate issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
axino or nick, can either of you attach an sosreport from an affected system? The crashdump doesn't include any userspace data so I can't see what exactly the qemu-nbd userspace program is doing, nor can i see what params it's started with. I'll need that info to be able to debug the qemu-nbd side of this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This memory leak we have so far only seen on arm64, to be clear. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This problem has caused more serious damage recently. When nbd dies and printk()s like mad, the serial console is not fast enough to display it. The kernel keeps allocating buffer space for serial output, which we see as 13G kmalloc-256 or kmalloc-512 kernel threads. Eventually the OOMkiller tries to free up space, but it can only kill userspace programs so ultimately the system dies altogether. This is more dire than mere CPU load or lockup warning messages. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Well, yes I agree, it does look like the serial port causing the softlockup is probably separate - but caused by - the nbd closed socket errors. However, the serial port output definitely shouldn't be causing a softlockup - no matter how much data it has to send, the serial port driver in the kernel should be scheduling itself during operation, so that it doesn't hog a single cpu for a long time. It's more likely that the general system "freezing" you are seeing is due to the serial port driver refusing to schedule off its cpu, and not any problem with the nbdX failure. I'll look into the nbd code also though, to see where that error is coming from and what that problem may be. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Re-reading comment #318 Dan, I realize that we may be investigating a symptom and not the root cause. Whenever the soft-lockup happens, the serial console does get flooded "block nbdX: Attempted send on closed socket". If the serial console getting flooded causes soft lockups, then it is indeed a concerning issue, but shouldn't we focus, in this bug, on making nbd not flood the console in the first place ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
> It turns out, the kernel booted through kexec fails booting probably because > of the notsc option : > https://pastebin.canonical.com/146714/ hmm, that's weird, but if notsc is all that changed i assume it is the problem. > I'm a bit worried about the following line : > [ 0.00] tsc: Kernel compiled with CONFIG_X86_TSC, cannot disable TSC > completely that's normal with notsc, the tsc is still there, it's just not used for the udelay function. but if it doesn't help the problem, no need to keep it. > I guess I can remove "notsc" from the kexec command line, but this will take > additional time. > I thought I'd let you know the current status in the meantime. ok thanks. I'll be out next week for the holidays, but continue looking at this Jan 1. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hi Dan, Thanks for your investigation. Sorry for the delay, but finally I managed to reboot the compute nodes with the "notsc" kernel parameter. I also disabled the qemu-nbd workaround. Once that was done, it didn't take long for a node to crash, which would indicate that notsc didn't fix the problem. However, the host got stuck and didn't dump anything. OK then. It happened a second time a few minutes after on a different host, so I thought I'd investigate this more. It turns out, the kernel booted through kexec fails booting probably because of the notsc option : https://pastebin.canonical.com/146714/ I'm a bit worried about the following line : [0.00] tsc: Kernel compiled with CONFIG_X86_TSC, cannot disable TSC completely which is also displayed during "regular" boots (eg not through kexec). I guess I can remove "notsc" from the kexec command line, but this will take additional time. I thought I'd let you know the current status in the meantime. Cheers -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Ok, here's my analysis of the latest dump. There are 3 kernel migrate threads waiting; this is the cause of the softlockup - specifically pid 101 on cpu 13 is where the softlockup (and then panic, due to panic on softlockup enabled) happens, and the other 2 migrate threads (pid 79 and 151) are also waiting. All are waiting for multi_cpu_stop to finish. The way multi_cpu_stop works is: the caller sets up one or more cpus to coordinate stopping; in multi_cpu_stop, the state machine moves from MULTI_STOP_PREPARE through disable irqs, to run (the provided function), to exit when done. However, only the specified cpus (in the cpumask) will run the function. The state machine doesn't proceed to the next step until all cpus have processed the current state. This is where the problem comes in. In this case, it's a migration of tasks from one numa node to another, via numa rebalancing. In this particular case, there are 3 rebalancing events happening: cpu 3 and cpu 10, cpu 3 and cpu 13, cpu 3 and cpu 20. the migrate threads on cpus 10, 13, and 20 are running multi_cpu_stop, but it's stuck waiting because cpu 3 still has it in its queue. cpu 3 is writing bytes to the serial port, and currently waiting for confirmation that the serial port write completed. This wait is done via checking the serial port register for CTS, then if it's not set delaying for 1us, and trying again. However, this is all inside a held spinlock, with irqs disabled. So while this serial port r/w is being done, nothing else will run on this cpu. But - the code limits this to 1 second, so presumably it shouldn't lock up the cpu for longer than 1 second or so (I haven't dug too far into this, so the function may be called multiple times with the lock held). For whatever reason, that serial port r/w seems to be taking a long time. The migrate threads on the other cpus are waiting for it to finish, so that the migrate thread on cpu 3 can run, and move the multi_cpu_stop state machine along. But that doesn't happen in time to avoid the softlockup detector. The multi_cpu_stop function could arguably use the addition of touch_nmi_watchdog(), since it intentionally spins on the cpu with interrupts disabled - doing so would avoid the softlockup detector (but would not change the system behavior). However, it's not really its fault, since the real cause is the other cpu(s) it's waiting for being locked. back on cpu 3 (that the others are waiting on), the way that delay is implemented is using the TSC. Unfortunately, the TSC is a generally unreliable clock source, so it's possible there is a problem in the delay function. To determine that, can you please boot with the "notsc" parameter, which will change the udelay function to use a simple loop instead of the TSC, and reproduce the softlockup? ** Changed in: linux (Ubuntu) Assignee: Rafael David Tinoco (inaddy) => Dan Streetman (ddstreet) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I've dl'ed the dump and I'm reviewing it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Junien, Sorry for the delay. After sometime dealing with some other priorities, I'm coming back to this. I'm downloading the dump and will take a look. Lets see what this bug is related with. Tks for providing it. Will report something back soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Unfortunately, one server managed to crashdump, even with your patched kernel. apport is above, crashdump is at https://private- fileshare.canonical.com/~axino/201511241217.crash.lp1505564.matar.tar.xz - sha1sum 056fae2554e52989a24094945b297c0c5906be7c I've diverted qemu-nbd again. Please let me know the next steps. Thanks ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport information ** Description changed: Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with: block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Nov 24 12:23 seq + crw-rw 1 root audio 116, 33 Nov 24 12:23 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.19 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 14.04 + IwConfig: Error: [Errno 2] No such file or directory + MachineType: HP ProLiant DL385 G7 + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen-256color + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 radeondrmfb + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-36-generic root=UUID=13289ac9-8dc9-4feb-b6bd-ca7db66b21d6 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M nox2apic intremap=off + ProcVersionSignature: Ubuntu 3.19.0-36.41~14.04.1hf00090138v20151122b1-generic 3.19.8-ckt9 + RelatedPackageVersions: + linux-restricted-modules-3.19.0-36-generic N/A + linux-backports-modules-3.19.0-36-generic N/A + linux-firmware 1.127.18 + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty uec-images + Uname: Linux 3.19.0-36-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + _MarkForUpload: True + dmi.bios.date: 02/02/2014 + dmi.bios.vendor: HP + dmi.bios.version: A18 + dmi.chassis.type: 23 + dmi.chassis.vendor: HP + dmi.modalias: dmi:bvnHP:bvrA18:bd02/02/2014:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: + dmi.product.name: ProLiant DL385 G7 + dmi.sys.vendor: HP ** Attachment added: "BootDmesg.txt" https://bugs.launchpad.net/bugs/1505564/+attachment/4524649/+files/BootDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Note to self: The commit being backported to 3.19 and 3.13 has to contain this race fix: commit dcc909d90ccdbb73226397ff6d298f7af35b0e11 Author: Markus Pargmann Date: Tue Oct 6 20:03:54 2015 +0200 nbd: Add locking for tasks The timeout handling introduced in 7e2893a16d3e (nbd: Fix timeout detection) introduces a race condition which may lead to killing of tasks that are not in nbd context anymore. This was not observed or reproducable yet. This patch adds locking to critical use of task_recv and task_send to avoid killing tasks that already left the NBD thread functions. This lock is only acquired if a timeout occures or the nbd device starts/stops. Reported-by: Ben Hutchings Signed-off-by: Markus Pargmann Reviewed-by: Ben Hutchings Fixes: 7e2893a16d3e ("nbd: Fix timeout detection") Signed-off-by: Jens Axboe Also. ** Patch removed: "trusty-nbd-restructure-debugging-prints.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524366/+files/trusty-nbd-restructure-debugging-prints.patch ** Patch removed: "trusty-nbd-fix-timeout-detection.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524367/+files/trusty-nbd-fix-timeout-detection.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Patch added: "trusty-nbd-restructure-debugging-prints.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524366/+files/trusty-nbd-restructure-debugging-prints.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Patch added: "trusty-nbd-fix-timeout-detection.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524367/+files/trusty-nbd-fix-timeout-detection.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Junien, That is good feedback. I also received another request to backport this to 3.13 SO I'll be providing the hotfixed kernel in the same PPA soon (tomorrow morning most likely). Attaching the 3.13 patches (just for reference since the SRU process requires me to send all those patches to kernel-team mailing list). Lets see if things continue good. If, by any chance, you are able to test this 3.13 kernel - maybe in another server - please provide me feedback also. Thank you very much Cheers Rafael Tinoco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hi Rafael, I applied the patch earlier today. No crash so far, which was nearly impossible before ! This looks very promising, I'll keep you posted tomorrow. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Okay, PPA is ready: https://launchpad.net/~inaddy/+archive/ubuntu/lp1505564/+packages Please upgrade kernel to: linux-lts-vivid - 3.19.0-36.41~14.04.1hf00090138v20151122b1 By doing: $ sudo add-apt-repository ppa:inaddy/lp1505564 $ sudo apt-get update $ sudo apt-get install linux-image-3.19.0-36-generic linux-image-extra-3.19.0-36-generic linux-headers-3.19.0-36-generic And make sure packages are being installed from PPA. Then reboot server using the hotfixed kernel. I'm looking forward on hearing feedback if this kernel mitigated the issues. Cheers Rafael Tinoco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hello Junien, Based on my previous feedbacks, I've created the following PPA: https://launchpad.net/~inaddy/+archive/ubuntu/lp1505564 With a Trusty HWE kernel (vivid) + 2 patches: nbd: Restructure debugging prints nbd: Fix timeout detection For you to use and provide me feedback. I've done minor tests and it looks like there are no regressions. Hopefully these patches will address the problem. If they do, I'll work on fixing Trusty, Vivid, Wily and Xenial. Cheers Rafael Tinoco PS: I'm still finishing kernel compilation and will copy packages to the PPA as soon as it is ready (it might take a few min/hours). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Testing patches I have attached above: inaddy@sf00090138trusty(~)$ sudo qemu-img create -f qcow2 -o preallocation=metadata ./test.qcow2 1G Formatting './test.qcow2', fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off inaddy@sf00090138trusty(~)$ sudo qemu-nbd --connect=/dev/nbd0 ./test.qcow2 [ 34.348125] nbd: registered device at major 43 [ 317.034493] nbd0: unknown partition table inaddy@sf00090138trusty(~)$ sudo fdisk /dev/nbd0 Device Boot Start End Blocks Id System /dev/nbd0p12048 2097151 1047552 83 Linux inaddy@sf00090138trusty(~)$ sudo mkfs.ext3 /dev/nbd0p1 inaddy@sf00090138trusty(~)$ sudo mkfs.ext3 /dev/nbd0p1 inaddy@sf00090138trusty(~)$ sudo mount /dev/nbd0p1 /mnt inaddy@sf00090138trusty(/mnt)$ dd if=/dev/zero of=./teste bs=1M count=256 oflag=direct 256+0 records in 256+0 records out 268435456 bytes (268 MB) copied, 1.15586 s, 232 MB/s Hopefully they won't cause any regression the PPA to be provided soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Patch added: "vivid-nbd-fix-timeout-detection.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4523782/+files/vivid-nbd-fix-timeout-detection.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Patch added: "vivid-nbd-restructure-debugging-prints.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4523781/+files/vivid-nbd-restructure-debugging-prints.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Description changed: - Hi, + Some of our nova compute hosts regularly freeze, sometimes for a few + hours, with kern.log getting spammed with: - Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket - and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : + and a few "CPU soft lockup" messages (see attached log). This clears up + when the queue gets cleared, eg : + block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. - - Note that timestamps from kern.log appears to be wrong, it looks like - the messages are being held, and then all delivered at once when the - kernel unfreezes. - - Attaching apport files from 2 hosts below. - --- - AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Oct 9 13:14 seq - crw-rw 1 root audio 116, 33 Oct 9 13:14 timer - AplayDevices: Error: [Errno 2] No such file or directory - ApportVersion: 2.14.1-0ubuntu3.15 - Architecture: amd64 - ArecordDevices: Error: [Errno 2] No such file or directory - AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: - DistroRelease: Ubuntu 14.04 - IwConfig: Error: [Errno 2] No such file or directory - MachineType: HP ProLiant DL385 G7 - Package: linux (not installed) - PciMultimedia: - - ProcEnviron: - TERM=screen-256color - PATH=(custom, no user) - LANG=en_US.UTF-8 - SHELL=/bin/bash - ProcFB: 0 radeondrmfb - ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash - ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 - RelatedPackageVersions: - linux-restricted-modules-3.19.0-30-generic N/A - linux-backports-modules-3.19.0-30-generic N/A - linux-firmware 1.127.15 - RfKill: Error: [Errno 2] No such file or directory - Tags: trusty uec-images - Uname: Linux 3.19.0-30-generic x86_64 - UpgradeStatus: No upgrade log present (probably fresh install) - UserGroups: - - _MarkForUpload: True - dmi.bios.date: 12/08/2012 - dmi.bios.vendor: HP - dmi.bios.version: A18 - dmi.chassis.type: 23 - dmi.chassis.vendor: HP - dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: - dmi.product.name: ProLiant DL385 G7 - dmi.sys.vendor: HP - --- - AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Oct 9 12:37 seq - crw-rw 1 root audio 116, 33 Oct 9 12:37 timer - AplayDevices: Error: [Errno 2] No such file or directory - ApportVersion: 2.14.1-0ubuntu3.15 - Architecture: amd64 - ArecordDevices: Error: [Errno 2] No such file or directory - AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: - DistroRelease: Ubuntu 14.04 - IwConfig: Error: [Errno 2] No such file or directory - MachineType: HP ProLiant DL360p Gen8 - Package: linux (not installed) - PciMultimedia: - - ProcEnviron: - TERM=screen-256color - PATH=(custom, no user) - LANG=en_US.UTF-8 - SHELL=/bin/bash - ProcFB: 0 VESA VGA - ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash - ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 - RelatedPackageVersions: - linux-restricted-modules-3.19.0-30-generic N/A - linux-backports-modules-3.19.0-30-generic N/A - linux-firmware 1.127.15 - RfKill: Error: [Errno 2] No such file or directory - StagingDrivers: visorchannel visorutil - Tags: trusty uec-images staging - Uname: Linux 3.19.0-30-generic x86_64 - UpgradeStatus: No upgrade log present (probably fresh install) - UserGroups: - - _MarkForUpload: True - dmi.bios.date: 11/14/2013 - dmi.bios.vendor: HP - dmi.bios.version: P71 - dmi.chassis.type: 23 - dmi.chassis.vendor: HP - dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: - dmi.product.name: ProLiant DL360p Gen8 - dmi.sys.vendor: HP - --- - AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Oct 28 13:07 seq - crw-rw 1 root audio 116, 33 Oct 28 13:07 timer - AplayDevices: Error: [Errno 2] No such file or directory - ApportVersion: 2.14.1-0ubuntu3.16 - Architecture: amd64 - ArecordDevices: Error: [Errno 2] No such file or directory - AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: - DistroRelease: Ubuntu 14.04 - IwConfig: Error: [Errno 2] No such file or directory - MachineType: HP ProLiant DL360p Gen8 - Package: linux (not installed) - PciMultimedia: - - ProcEnviron: - TERM=screen-256color - PATH=(custom, no user) - LANG=en_US.UTF-8 - SHELL=/bin/bash - ProcFB: 0 VESA VGA - ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Junien, I faced minor issues on backport yesterday and today is holiday in Brazil. I'll get back to this soon. Nevertheless, it is good feedback that this "qemu-nbd" workaround is probably making the system more stable. I'll get back to you soon. Thank you Rafael -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hi Rafael, WIth qemu-nbd symlinked to /bin/true, no crash so far... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Thanks for your update Rafael. Since nova-compute doesn't do anything useful with qemu-nbd anyway, I'm going to try to "soft-disable" it (divert + symlink to /bin/true), and we'll see if we can repro the crashes. I'll keep you posted. I'll also try your patched kernel as soon as it's ready, of course :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hello Junien, After your last crash - similar to previous ones - one thing called my attention: For the first time we had one CPU RCU stall detected by another CPU. This made me think that it wasn't only related to the SMP logic - like I believed - but the stall occurred also somewhere else. [ 5792.466770] INFO: rcu_sched detected stalls on CPUs/tasks: { 7} (detected by 15, t=15003 jiffies, g=182379, c=182378, q=0) And this stall happened before Async I/O callbacks started to be suppressed: [ 5793.190218] block nbd6: Attempted send on closed socket [ 5793.190221] blk_update_request: 1154 callbacks suppressed [ 5793.190223] blk_update_request: I/O error, dev nbd6, sector 125828992 [ 5793.190226] buffer_io_error: 1151 callbacks suppressed [ 5793.190227] Buffer I/O error on dev nbd6, logical block 125828992, async page read [ 5793.190235] block nbd6: Attempted send on closed socket [ 5793.190237] blk_update_request: I/O error, dev nbd6, sector 125828993 [ 5793.190238] Buffer I/O error on dev nbd6, logical block 125828993, async page read [ 5793.190242] block nbd6: Attempted send on closed socket [ 5793.190243] blk_update_request: I/O error, dev nbd6, sector 125828994 [ 5793.190245] Buffer I/O error on dev nbd6, logical block 125828994, async page read [ 5793.190248] block nbd6: Attempted send on closed socket Digging upstream (from 3.13 to HEAD) I could see there were not a huge amount of fixes: $ git log --pretty=oneline v3.13..HEAD -- drivers/block/nbd.c | wc -l 31 For nbd.c and I identified an improvement on nbd timeout handling: commit 7e2893a16d3e71035a38122a77bc55848a29f0e4 Author: Markus Pargmann Date: Mon Aug 17 08:20:00 2015 +0200 nbd: Fix timeout detection This fix is pretty recent (4.3) and it fit to the case: 3.18 kernel facing the same issue. Later I found out that Debian had a similar bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770479 https://lists.debian.org/debian-kernel/2015/05/msg00054.html for kernel 3.16, complaining about messages like this: [ 5793.190242] block nbd6: Attempted send on closed socket And the lack of proper timeout for nbd connections (now based on timeout after IO submission). SO... The backport shall be easy* and I'll probably make one PPA containing a 3.18 (+ this patch) available for you tomorrow. * 2 out of 12 hunks FAILED -- saving rejects to file drivers/block/nbd.c.rej * Debian has a 3.16 version already Thank you Rafael Tinoco ** Bug watch added: Debian Bug tracker #770479 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770479 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Thank Junien, I'm downloading the crash dump (10GB) and will update you as soon as I open it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Tags added: kernel-key ** Changed in: linux (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport above, crash dump is at https://chinstrap.canonical.com/~axino/201511171222.crash.lp1505564.druk.tar.xz - sha1sum 93ae006186b6bc7298afd37d3f759effe08d7ba3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport information ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 28 13:07 seq crw-rw 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-restricted-modules-3.13.0
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hi Rafael, For starters, the server Chris mentioned above didn't panic because the kernel.softlockup_panic wasn't set to 1 on reboot. This is now fixed. Then, we're still running 3.19 (all the nodes got rebooted to 3.19.0-33-generic). Let me know if you wish us to get back to 3.13. I verified that all the firmwares were the most recent ones, and they were. I rebooted all the nodes with the proper x2apic kernel options. I also disabled all C-States, and also set everything relevant to "performance". You can see the changes here : http://paste.ubuntu.com/13312776/ (this paste is showing all possible settings in G7 and Gen8, I of course could only apply the settings that existed on each infrastructure). Unfortunately, even with all this, we had a G7 that panic'ed and crashdump'ed about ~1h after I set it back in the compute pool. You will find the apport and crashdump below. Let me know what are the next steps. Thanks ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hello Chris, Could you clarify the following statement: """ So, step 2 was to add "nox2apic intremap=off" to the DL385-G7s. I added it to only one of them initially. That machine lasted 9 days before we had another kernel panic ("NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [migration/27:200]"), but after the panic it seems to have settled back down again (without any reboot). """ So, I'm not sure if you are "panic'ing on hung tasks" (sysctl option). The way I read this is that the machine showed a soft lockup BUT the kernel did not crash and recovered after some time. This might indicate that, after workload was reduced, the kernel could get back on track with migration kthread. Could you clarify this ? You did right. < G8 cmdline == "nox2apic intremap=off" >= G8 cmdline == "intremap=no_x2apic_optout" So, if the kernel (G7) had a soft lockup warning but had no "hard lockups" (race conditions), then we are good. Judging by the G8, it looks like that after the change it is still running. Could you clarify if you changed the c-states (min and packing) firmware options ? I would recommend you staying in 3.13 if they show stable after firmware version/options and cmdline were changed. This way we have a way to "compare" things. As long as they don't have HARD lockups, I think we will be good. Let me know if you need any other clarification. Cheers! Rafael Tinoco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hi Rafael, I've been continuing Junien's investigations into this problem. The machines have had all the BIOS and firmware updates I could find on HP's website (although in the case of a DL385-G7 the latest appears to be February 2014!) One of them only lasted a day before crashing again. So, step 2 was to add "nox2apic intremap=off" to the DL385-G7s. I added it to only one of them initially. That machine lasted 9 days before we had another kernel panic ("NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [migration/27:200]"), but after the panic it seems to have settled back down again (without any reboot). I've also added "intremap=no_x2apic_optout" to one of the DL360-G8s after it crashed a couple of days ago. So far, it's doing ok. I''m tempted to try upgrading them to linux-image-generic-lts-wily (currently 4.2.0.18.13) unless there's any information from the current setup that could be useful. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Hello Junien, (recommendations with *) I'm replying to you and to the LP bug so it gets proper documentation. Under comment #91: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/comments/91 You can see my kernel dump analysis, where I am showing you that the OS is stuck in a "migration thread", possibly because of a lack of IPIs synchronisation (maybe even an IPI being lost). We have already seen cases like this - specially in nested virtualisation environments - and this has been discussed in LKML. Before we move further I need you to follow some kind of "best practices" for Proliant Servers: 1 - NMIs caused during MWAIT instruction (caused by intel_idle module): & HP Proliant Servers - Kernel Panic - NMI - DL360 & DL380 - HPWDT module loaded (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1417580) (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1432837) * Firmware: Configure a maximum of a C3 c-state for CPU savings (CPU C-STATES) * Firmware: Disable packed CPU c-state * Firmware: Disable Cooperative Power Management * Make sure NOT TO LOAD HPWDT kernel module (LP: #1432837 Fix Released 3.13.0-49.81) 2 - Recently discovered NMIs caused by a BUG in Intel microcode (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1416414) ** If you have Intel based Proliant Servers, because of Intel microcode issue, use at least* 3.13.0-35.61. 3 - X2APIC support for HP Proliant Servers (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1398497) * For Proliant prior to G8 (<= G7) use "nox2apic intremap=off" into grub cmdline * For Proliant G8 use "intremap=no_x2apic_optout" into grub cmdline 4 - HP Proliant Latest Firmware MOST IMPORTANT Upgrade server firmware to latest version There were numerous firmware fixes from HP. ---> If we are facing a firmware problem - related to IPIs, the inter-processor-interrupts, being missed - we have to make sure this is reproducible in the latest firmware in order to work together with HP ROM engineering team. Summary: Could you follow all these steps and provide feedback ? I understand this might take awhile if you have a big number of servers and - if so - I would take a statistical approach here, by changing only half of the servers and sticking with the first half as the "control group", for future comparisons. Is this feasible ? Looking forward to hearing your feedback. Best Regards Rafael Tinoco Sustaining Engineering -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Changed in: linux (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Attachment added: "lp1505564-3.13-kdump-crash-output.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4509474/+files/lp1505564-3.13-kdump-crash-output.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I'm attaching the crash tool output from the 3.13 kernel dump. Much likely related to the situation already found in the following case: -> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540 Handled by Chris Arges and I on LKML discussions with Ingo and Linus: -> http://www.kernelhub.org/?p=2&msg=683682 FOR NOW, it is LIKELY that I'll rely on already known recommendations for Proliant (including the ones related to X2APIC mode): -> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1417580 So we can TRY TO GUARANTEE that there are no LOST IRQs (IPIs) using the firmware you're using. Hopefully with the proper APIC mode set, like HP recommends, we will not have those IPIs problems. OBS: Whenever IPIs are lost (we've seen this on some nested KVMs and some buggy HW) we can be locked up in the SMP callback state machine. This means that the state machine looses IPIs ACKs and the state machine loops forever trying to shutdown the CPU for the SMP task queue to continue. I'll provide SOON a comment with SUGGESTIONS and asking for FEEDBACK. For now, from the 3.13 kernel dump, the most interesting part: We had 7 CPUs executing the migration kernel thread (for the SMP callback state machine execution): migration tasks (state machine loop) >93 2 4 8808147b47d0 RU 0.0 0 0 [migration/4] > 118 2 9 881814a2c7d0 RU 0.0 0 0 [migration/9] > 123 2 10 88081404c7d0 RU 0.0 0 0 [migration/10] > 128 2 11 881814a4c7d0 RU 0.0 0 0 [migration/11] > 138 2 13 881814a647d0 RU 0.0 0 0 [migration/13] > 165 2 18 8810149ec7d0 RU 0.0 0 0 [migration/18] > 195 2 24 881014a647d0 RU 0.0 0 0 [migration/24] This logic will try to migrate tasks from one CPU to another. In order for that to happen they have to rely on the state machine logic of shutting CPUs down before migrating the tasks (turning off IRQs, etc). The state machine - shutting down the CPUs on phases - relies on the SMP callbacks bellow. We had 3 CPUs in a part of the kernel that we have already identified to be problematic under certain conditions and/or HW. ** > 17247 1 23 881007055fc0 RU 1.6 7358428 2192548 qemu- system-x86 PID: 17247 TASK: 881007055fc0 CPU: 23 COMMAND: "qemu-system-x86" #0 [88203eac6e58] crash_nmi_callback at 8103fb72 #1 [88203eac6e68] nmi_handle at 8171f188 #2 [88203eac6ec8] do_nmi at 8171f350 #3 [88203eac6ef0] end_repeat_nmi at 8171e5f1 [exception RIP: generic_exec_single+130] RIP: 810db712 RSP: 8810ea7c96e0 RFLAGS: 0202 RAX: 0010 RBX: 0010 RCX: 0202 RDX: 8810ea7c96e0 RSI: 0018 RDI: 0001 RBP: 810db712 R8: 810db712 R9: 0018 R10: 8810ea7c96e0 R11: 0202 R12: R13: 0206 R14: 7bc87bc6 R15: 8814959f76c0 ORIG_RAX: 8814959f76c0 CS: 0010 SS: 0018 --- --- #4 [8810ea7c96e0] generic_exec_single at 810db712 CSD_FLAG logic discussed with Linus 108 while (csd->flags & CSD_FLAG_LOCK) 0x810db712 <+130>: testb $0x1,0x20(%rbx) 0x810db716 <+134>: jne0x810db710 109 cpu_relax(); 110 } ** > 21036 1 27 8810b69947d0 RU 1.0 7484828 1401940 qemu- system-x86 PID: 21036 TASK: 8810b69947d0 CPU: 27 COMMAND: "qemu-system-x86" #0 [88203eb46e58] crash_nmi_callback at 8103fb72 #1 [88203eb46e68] nmi_handle at 8171f188 #2 [88203eb46ec8] do_nmi at 8171f350 #3 [88203eb46ef0] end_repeat_nmi at 8171e5f1 [exception RIP: generic_exec_single+130] RIP: 810db712 RSP: 8814959f7670 RFLAGS: 0202 RAX: 0010 RBX: 0010 RCX: 0202 RDX: 8814959f7670 RSI: 0018 RDI: 0001 RBP: 810db712 R8: 810db712 R9: 0018 R10: 8814959f7670 R11: 0202 R12: R13: 0282 R14: R15: 0100 ORIG_RAX: 0100 CS: 0010 SS: 0018 --- --- #4 [8814959f7670] generic_exec_single at 810db712 CSD_FLAG logic discussed with Linus 108 while (csd->flags & CSD_FLAG_LOCK) 0x810db712 <+130>: testb $0x1,0x20(%rbx) 0x810db716 <+134>: jne0x810db710 109 cpu_relax(); 110 } ** > 18516 1 31 881dd54a2fe0 RU 1.6 7358428 2192548 qemu- system-x86 PID: 18516 TASK: 881dd54a2fe0 CPU: 31 COMMAND: "qemu-system-x86" #0 [88203ebc6e58] crash_nmi_callback at 8103fb72 #1 [88203ebc6e68] nmi_handle at fff
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I'm attaching the crash tool output from the 3.13 kernel dump. Much likely related to the situation already found in the following case: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540 Handled by Chris Arges and I on LKML discussions with Ingo and Linus: -> http://www.kernelhub.org/?p=2&msg=683682 FOR NOW, it is LIKELY that I'll rely on already known recommendations for Proliant (including the ones related to X2APIC mode): https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1417580 So we can TRY TO GUARANTEE that there are no LOST IRQs (IPIs) using the firmware you're using. Hopefully with the proper APIC mode set, like HP recommends, we will not have those IPIs problems. OBS: Whenever IPIs are lost (we've seen this on some nested KVMs and some buggy HW) we can be locked up in the SMP callback state machine. This means that the state machine looses IPIs ACKs and the state machine loops forever trying to shutdown the CPU for the SMP task queue to continue. I'll provide SOON a comment with SUGGESTIONS and asking for FEEDBACK. For now, from the 3.13 kernel dump, the most interesting part: We had 7 CPUs executing the migration kernel thread (for the SMP callback state machine execution): migration tasks (state machine loop) >93 2 4 8808147b47d0 RU 0.0 0 0 [migration/4] > 118 2 9 881814a2c7d0 RU 0.0 0 0 [migration/9] > 123 2 10 88081404c7d0 RU 0.0 0 0 [migration/10] > 128 2 11 881814a4c7d0 RU 0.0 0 0 [migration/11] > 138 2 13 881814a647d0 RU 0.0 0 0 [migration/13] > 165 2 18 8810149ec7d0 RU 0.0 0 0 [migration/18] > 195 2 24 881014a647d0 RU 0.0 0 0 [migration/24] This logic will try to migrate tasks from one CPU to another. In order for that to happen they have to rely on the state machine logic of shutting CPUs down before migrating the tasks (turning off IRQs, etc). The state machine - shutting down the CPUs on phases - relies on the SMP callbacks bellow. We had 3 CPUs in a part of the kernel that we have already identified to be problematic under certain conditions and/or HW. ** > 17247 1 23 881007055fc0 RU 1.6 7358428 2192548 qemu- system-x86 PID: 17247 TASK: 881007055fc0 CPU: 23 COMMAND: "qemu-system-x86" #0 [88203eac6e58] crash_nmi_callback at 8103fb72 #1 [88203eac6e68] nmi_handle at 8171f188 #2 [88203eac6ec8] do_nmi at 8171f350 #3 [88203eac6ef0] end_repeat_nmi at 8171e5f1 [exception RIP: generic_exec_single+130] RIP: 810db712 RSP: 8810ea7c96e0 RFLAGS: 0202 RAX: 0010 RBX: 0010 RCX: 0202 RDX: 8810ea7c96e0 RSI: 0018 RDI: 0001 RBP: 810db712 R8: 810db712 R9: 0018 R10: 8810ea7c96e0 R11: 0202 R12: R13: 0206 R14: 7bc87bc6 R15: 8814959f76c0 ORIG_RAX: 8814959f76c0 CS: 0010 SS: 0018 --- --- #4 [8810ea7c96e0] generic_exec_single at 810db712 CSD_FLAG logic discussed with Linus 108 while (csd->flags & CSD_FLAG_LOCK) 0x810db712 <+130>: testb $0x1,0x20(%rbx) 0x810db716 <+134>: jne0x810db710 109 cpu_relax(); 110 } ** > 21036 1 27 8810b69947d0 RU 1.0 7484828 1401940 qemu- system-x86 PID: 21036 TASK: 8810b69947d0 CPU: 27 COMMAND: "qemu-system-x86" #0 [88203eb46e58] crash_nmi_callback at 8103fb72 #1 [88203eb46e68] nmi_handle at 8171f188 #2 [88203eb46ec8] do_nmi at 8171f350 #3 [88203eb46ef0] end_repeat_nmi at 8171e5f1 [exception RIP: generic_exec_single+130] RIP: 810db712 RSP: 8814959f7670 RFLAGS: 0202 RAX: 0010 RBX: 0010 RCX: 0202 RDX: 8814959f7670 RSI: 0018 RDI: 0001 RBP: 810db712 R8: 810db712 R9: 0018 R10: 8814959f7670 R11: 0202 R12: R13: 0282 R14: R15: 0100 ORIG_RAX: 0100 CS: 0010 SS: 0018 --- --- #4 [8814959f7670] generic_exec_single at 810db712 CSD_FLAG logic discussed with Linus 108 while (csd->flags & CSD_FLAG_LOCK) 0x810db712 <+130>: testb $0x1,0x20(%rbx) 0x810db716 <+134>: jne0x810db710 109 cpu_relax(); 110 } ** > 18516 1 31 881dd54a2fe0 RU 1.6 7358428 2192548 qemu- system-x86 PID: 18516 TASK: 881dd54a2fe0 CPU: 31 COMMAND: "qemu-system-x86" #0 [88203ebc6e58] crash_nmi_callback at 8103fb72 #1 [88203ebc6e68] nmi_handle at
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Junien, I'm on it right now.. will update here asap. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Yet another crash, on another node this time (still a 100% Nova compute node). apport information is above, crashdump is at https://chinstrap.canonical.com/~axino/201510301227.crash.lp1505564.phianna.tar.xz - sha1sum 71353f8c70d009369a61de811c90d6199b341543 Thanks ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport information ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 28 13:07 seq crw-rw 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-restricted-modules-3.13.0
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
crashdump available at https://chinstrap.canonical.com/~axino/201510292103.crash.lp1505564.orlo.tar.xz - sha1sum is 366c0460cceed5938f2a19fc4b925380a33c18a6 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Upgraded all the kernels to lts-vivid (3.19.0-31-generic), and got a new crashdump overnight, from the same server just above (orlo). apport + dump below. ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 28 13:07 seq crw-rw 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkern
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
sha1 sums for all 3 dumps below : 6b63d74566b6df0671ba9e79dca724ddc6d8d6df 201510281259.crash.lp1505564.tar.xz <= may have been taken after the lockup occured 3a8cbdd9e51af4f6eaba4ff0aacc6f956c706961 201510281618.crash.lp1505564.druk.tar.xz 1ebd57dea13cf655e7ef442951da2aedc33d0046 201510281951.crash.lp1505564.orlo.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
dump at https://chinstrap.canonical.com/~axino/201510281951.crash.lp1505564.orlo.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Second apport+dump below ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 28 13:07 seq crw-rw 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-restricted-modules-
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
dump is at https://chinstrap.canonical.com/~axino/201510281618.crash.lp1505564.druk.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I'm just now realizing that the crashdump above may have been taken too late (when the kernel wasn't locked up anymre), because I could ssh to the server when I took it. I was seeing the "block nbdX: Attempted send on closed socket" kernel log spam on the serial when I sent the NMI, but _perhaps_ these messages were just earlier messages that the serial was still catching up with. Anyway, I got 2 new dumps, and these 2 were triggered automaticallt by kernel.softlockup_panic so they might be better. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
First dump + apport (post reboot) below ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 28 13:07 seq crw-rw 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-rest
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
This issue just hit us again, this time I sent an NMI to the server to get a dump. It's available at https://chinstrap.canonical.com/~axino/201510281259.crash.lp1505564.tar.xz apport information post-reboot is available above. We've been trying to see if the issue appeared somewhere in the 3.13 series, hence the 3.13.0-29-generic kernel version. Thanks ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport information ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 12:37 seq crw-rw 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Oct 28 13:07 seq + crw-rw 1 root audio 116, 33 Oct 28 13:07 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.16 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 14.04 + IwConfig: Error: [Errno 2] No such file or directory + MachineType: HP ProLiant DL360p Gen8 + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen-256color + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M + ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 + RelatedPackageVersions: + linux-restricted-modules-3.13.0
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
** Changed in: linux (Ubuntu) Assignee: (unassigned) => Rafael David Tinoco (inaddy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
I think that this may be a duplicate of #1500739, the symptoms certainly look the same. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
Second host now ** Tags added: staging ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 9 13:14 seq crw-rw 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Oct 9 12:37 seq + crw-rw 1 root audio 116, 33 Oct 9 12:37 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.15 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 14.04 + IwConfig: Error: [Errno 2] No such file or directory + MachineType: HP ProLiant DL360p Gen8 + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen-256color + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash + ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 + RelatedPackageVersions: + linux-restricted-modules-3.19.0-30-generic N/A + linux-backports-modules-3.19.0-30-generic N/A + linux-firmware 1.127.15 + RfKill: Error: [Errno 2] No such file or directory + StagingDrivers: visorchannel visorutil + Tags: trusty uec-images staging + Uname: Linux 3.19.0-30-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + _MarkForUpload: True + dmi.bios.date: 11/14/2013 + dmi.bios.vendor: HP + dmi.bios.version: P71 + dmi.chassis.type: 23 + dmi.chassis.vendor: HP + dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: + dmi.product.name: ProLiant DL360p Gen8 + dmi.sys.vendor: HP -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1505564] Re: Soft lockup with "block nbdX: Attempted send on closed socket" spam
apport information ** Tags added: apport-collected trusty uec-images ** Description changed: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Oct 9 13:14 seq + crw-rw 1 root audio 116, 33 Oct 9 13:14 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.15 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 14.04 + IwConfig: Error: [Errno 2] No such file or directory + MachineType: HP ProLiant DL385 G7 + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen-256color + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 radeondrmfb + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash + ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 + RelatedPackageVersions: + linux-restricted-modules-3.19.0-30-generic N/A + linux-backports-modules-3.19.0-30-generic N/A + linux-firmware 1.127.15 + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty uec-images + Uname: Linux 3.19.0-30-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + _MarkForUpload: True + dmi.bios.date: 12/08/2012 + dmi.bios.vendor: HP + dmi.bios.version: A18 + dmi.chassis.type: 23 + dmi.chassis.vendor: HP + dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: + dmi.product.name: ProLiant DL385 G7 + dmi.sys.vendor: HP ** Attachment added: "BootDmesg.txt" https://bugs.launchpad.net/bugs/1505564/+attachment/4493172/+files/BootDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs