apport information ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1576599/+attachment/4651225/+files/CurrentDmesg.txt
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1576599 Title: XFS Oopsing 4.2.0-35-generic Kernel - Block out of range Status in linux package in Ubuntu: Confirmed Bug description: After updating the Kernel on two of our Trusty Ceph servers from 3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers have now Oopsed multiple times accessing some of their XFS filesystems. Once this error occurs, the XFS filesystem becomes permanently inaccessible. Even after a reboot, any attempt to access the filesystem in question either by mounting it, or attempting an xfs_repair will trigger the same Oops again. These filesystems did not immediately fail until several hours after the upgrade. I suspect that this is related to some housekeeping being triggered within the filesystem. This is the call trace from the moment a filesystem first failed: Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------ Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]() Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G D W 4.2.0-35-generic #40~14.04.1-Ubuntu Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014 Apr 28 22:47:25 ceph-store5 kernel: [26692.911303] 0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000 Apr 28 22:47:25 ceph-store5 kernel: [26692.919677] ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508 Apr 28 22:47:25 ceph-store5 kernel: [26692.928033] ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8 Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace: Apr 28 22:47:25 ceph-store5 kernel: [26692.939208] [<ffffffff817bcbf8>] dump_stack+0x63/0x81 Apr 28 22:47:25 ceph-store5 kernel: [26692.944992] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0 Apr 28 22:47:25 ceph-store5 kernel: [26692.951752] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20 Apr 28 22:47:25 ceph-store5 kernel: [26692.958346] [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26692.965221] [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26692.972194] [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26692.979763] [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26692.987412] [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26692.994564] [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.002302] [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.009379] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.016229] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10 Apr 28 22:47:25 ceph-store5 kernel: [26693.022814] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.029669] [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.036915] [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.044074] [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.050833] [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.057882] [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.065403] [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280 Apr 28 22:47:25 ceph-store5 kernel: [26693.071509] [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.078078] [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.085129] [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.093151] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.100199] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.107251] [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.114009] [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.120647] [<ffffffff81212a42>] generic_setxattr+0x62/0x80 Apr 28 22:47:25 ceph-store5 kernel: [26693.127107] [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0 Apr 28 22:47:25 ceph-store5 kernel: [26693.134045] [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0 Apr 28 22:47:25 ceph-store5 kernel: [26693.145904] [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0 Apr 28 22:47:25 ceph-store5 kernel: [26693.151781] [<ffffffff812137fa>] setxattr+0x12a/0x1b0 Apr 28 22:47:25 ceph-store5 kernel: [26693.157626] [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.164599] [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.172451] [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.179694] [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0 Apr 28 22:47:25 ceph-store5 kernel: [26693.186278] [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs] Apr 28 22:47:25 ceph-store5 kernel: [26693.193122] [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0 Apr 28 22:47:25 ceph-store5 kernel: [26693.199193] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75 Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]--- Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------ I notice that xfs_da_grow_inode. Am I right in thinking that this was a new feature add in the 4.2 kernel? I expect that the history of these filesystems is important. These would more have been formatted on a 3.16 kernel. Although I can't inspect these failed filesystems, I can give information about the other filesystems on the same servers, which I an 99% sure are identical: root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35 meta-data=/dev/bcache6 isize=512 agcount=4, agsize=244188661 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1 data = bsize=4096 blocks=976754644, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=476930, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 ==== Attempt to repair a filesystem (after a reboot) ==== root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/ Killed Dmesg output (dmesg -T): [Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem [Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal) [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------ [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]() [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014 [Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000 [Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928 [Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18 [Fri Apr 29 10:18:49 2016] Call Trace: [Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81 [Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0 [Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20 [Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210 [Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10 [Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0 [Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160 [Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120 [Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0 [Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40 [Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150 [Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0 [Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75 [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]--- [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------ [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]() [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014 [Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000 [Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928 [Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18 [Fri Apr 29 10:18:49 2016] Call Trace: [Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81 [Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0 [Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20 [Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210 [Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10 [Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0 [Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160 [Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120 [Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0 [Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40 [Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150 [Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0 [Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75 [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]--- [Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8 [Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs] [Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0 [Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014 [Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000 [Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>] [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs] [Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08 EFLAGS: 00010292 [Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52 [Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828 [Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0 [Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828 [Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008 [Fri Apr 29 10:18:49 2016] FS: 00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000 [Fri Apr 29 10:18:49 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0 [Fri Apr 29 10:18:49 2016] Stack: [Fri Apr 29 10:18:49 2016] ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800 [Fri Apr 29 10:18:49 2016] ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000 [Fri Apr 29 10:18:49 2016] ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000 [Fri Apr 29 10:18:49 2016] Call Trace: [Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210 [Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10 [Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0 [Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs] [Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160 [Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120 [Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0 [Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40 [Fri Apr 29 10:18:50 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150 [Fri Apr 29 10:18:50 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0 [Fri Apr 29 10:18:50 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75 [Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b [Fri Apr 29 10:18:50 2016] RIP [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs] [Fri Apr 29 10:18:50 2016] RSP <ffff88018e11ba08> [Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8 [Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]--- root@ceph-store5:~# Thanks to the magic of Ceph, I've so far managed to avoid data-loss and serious downtime, but I suspect there's a timebomb here waiting for anyone who upgrades from an older kernel to 4.2.0+. Please let me know is there's any further information I might be able to provide to track this down. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Apr 29 01:29 seq crw-rw---- 1 root audio 116, 33 Apr 29 01:29 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.19 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 MachineType: Dell Inc. PowerEdge R720xd Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-35-generic root=UUID=f022f2e6-eeba-4e27-b286-6acd2c4bd1ea ro net.ifnames=1 biosdevname=0 console=tty1 console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/di-initrd BOOTIF=01-b0-83-fe-e6-17-cc nomdmonddf nomdmonisw crashkernel=384M-:128M ProcVersionSignature: Ubuntu 4.2.0-35.40~14.04.1-generic 4.2.8-ckt5 RelatedPackageVersions: linux-restricted-modules-4.2.0-35-generic N/A linux-backports-modules-4.2.0-35-generic N/A linux-firmware 1.127.22 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 4.2.0-35-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 07/09/2014 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.4.3 dmi.board.name: 0HJK12 dmi.board.vendor: Dell Inc. dmi.board.version: A03 dmi.chassis.asset.tag: 707 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.4.3:bd07/09/2014:svnDellInc.:pnPowerEdgeR720xd:pvr:rvnDellInc.:rn0HJK12:rvrA03:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R720xd dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1576599/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
