ENOSPC / no space on very large devices

2016-07-19 Thread Stefan Priebe - Profihost AG
Hi list,

while i didn't had the problem for some month i'm now getting ENOSPC on
a regular basis on one host.

It would be great if someone can help me debugging this.

Some basic informations:
# touch /vmbackup/abc
touch: cannot touch `/vmbackup/abc': No space left on device

# df -h /vmbackup/
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/stripe0-vmbackup   37T   28T  8,5T  77% /vmbackup

# btrfs filesystem df /vmbackup/
Data, single: total=27.87TiB, used=27.39TiB
System, DUP: total=8.00MiB, used=4.34MiB
Metadata, DUP: total=286.50GiB, used=199.91GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs filesystem show /vmbackup/
Label: none  uuid: c8c3abf7-8280-4baa-bb51-a8c599e48002
Total devices 1 FS bytes used 27.59TiB
devid1 size 36.38TiB used 28.43TiB path
/dev/mapper/stripe0-vmbackup

# mount | grep vmbackup
/dev/mapper/stripe0-vmbackup on /vmbackup type btrfs
(rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,clear_cache,commit=300,subvolid=5,subvol=/)

dmesg is empty.

if i umount the volume i get traces (i already did a clear_cache 4 days
ago to recalculate the space_tree):

[545031.675797] [ cut here ]
[545031.725166] WARNING: CPU: 1 PID: 17711 at
fs/btrfs/extent-tree.c:5710 btrfs_free_block_groups+0x35a/0x400 [btrfs]()
[545031.778329] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandler
button btrfs dm_mod raid1 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq md_mod ixgbe mdio sg sd_mod
ahci ptp libahci megaraid_sas pps_core
[545032.081037] CPU: 1 PID: 17711 Comm: umount Tainted: G   O
4.4.15+43-ph #1
[545032.145078] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
02/18/2015
[545032.210238]   88010c40bcd8 bd3c712f

[545032.275650]  c03ec603 88010c40bd18 bd0837e7
0047a000
[545032.341525]   88105e0ea400 881054a76088
881054a76000
[545032.408500] Call Trace:
[545032.475272]  [] dump_stack+0x63/0x84
[545032.543620]  [] warn_slowpath_common+0x97/0xe0
[545032.612900]  [] warn_slowpath_null+0x1a/0x20
[545032.682026]  []
btrfs_free_block_groups+0x35a/0x400 [btrfs]
[545032.750297]  [] close_ctree+0x15b/0x330 [btrfs]
[545032.817085]  [] btrfs_put_super+0x19/0x20 [btrfs]
[545032.883439]  [] generic_shutdown_super+0x6f/0x100
[545032.949302]  [] kill_anon_super+0x16/0x30
[545033.014327]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
[545033.079031]  [] deactivate_locked_super+0x51/0x90
[545033.143275]  [] deactivate_super+0x4e/0x70
[545033.206535]  [] cleanup_mnt+0x43/0x90
[545033.268842]  [] __cleanup_mnt+0x12/0x20
[545033.331629]  [] task_work_run+0x7e/0xa0
[545033.393350]  [] exit_to_usermode_loop+0x66/0x95
[545033.454685]  [] syscall_return_slowpath+0xa6/0xf0
[545033.515485]  [] int_ret_from_sys_call+0x25/0x8f
[545033.575890] ---[ end trace bd985b05cc90617c ]---
[545033.636708] [ cut here ]
[545033.696339] WARNING: CPU: 1 PID: 17711 at
fs/btrfs/extent-tree.c:5711 btrfs_free_block_groups+0x3f4/0x400 [btrfs]()
[545033.758031] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandler
button btrfs dm_mod raid1 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq md_mod ixgbe mdio sg sd_mod
ahci ptp libahci megaraid_sas pps_core
[545034.095188] CPU: 1 PID: 17711 Comm: umount Tainted: GW  O
4.4.15+43-ph #1
[545034.166070] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
02/18/2015
[545034.236259]   88010c40bcd8 bd3c712f

[545034.307690]  c03ec603 88010c40bd18 bd0837e7
0047a000
[545034.379596]   88105e0ea400 881054a76088
881054a76000
[545034.452542] Call Trace:
[545034.525286]  [] dump_stack+0x63/0x84
[545034.599643]  [] warn_slowpath_common+0x97/0xe0
[545034.674894]  [] warn_slowpath_null+0x1a/0x20
[545034.750338]  []
btrfs_free_block_groups+0x3f4/0x400 [btrfs]
[545034.826354]  [] close_ctree+0x15b/0x330 [btrfs]
[545034.900758]  [] btrfs_put_super+0x19/0x20 [btrfs]
[545034.973612]  [] generic_shutdown_super+0x6f/0x100
[545035.044589]  [] kill_anon_super+0x16/0x30
[545035.113505]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
[545035.180769]  [] deactivate_locked_super+0x51/0x90
[545035.246451]  [] deactivate_super+0x4e/0x70
[545035.311231]  [] cleanup_mnt+0x43/0x90
[545035.374958]  [] __cleanup_mnt+0x12/0

Re: ENOSPC / no space on very large devices

2016-07-19 Thread Wang Xiaoguang

hello,

On 07/20/2016 01:31 PM, Stefan Priebe - Profihost AG wrote:

Hi list,

while i didn't had the problem for some month i'm now getting ENOSPC on
a regular basis on one host.

It would be great if someone can help me debugging this.

Some basic informations:
# touch /vmbackup/abc
touch: cannot touch `/vmbackup/abc': No space left on device

When touch operation failed, would you please change dir to
/sys/fs/btrfs/UUID/allocation/data/ and show me these files' value.
And also files in /sys/fs/btrfs/UUID/allocation/metadata. thanks.
Here UUID is your real uuid :)

Regards,
Xiaoguang Wang


# df -h /vmbackup/
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/stripe0-vmbackup   37T   28T  8,5T  77% /vmbackup

# btrfs filesystem df /vmbackup/
Data, single: total=27.87TiB, used=27.39TiB
System, DUP: total=8.00MiB, used=4.34MiB
Metadata, DUP: total=286.50GiB, used=199.91GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs filesystem show /vmbackup/
Label: none  uuid: c8c3abf7-8280-4baa-bb51-a8c599e48002
 Total devices 1 FS bytes used 27.59TiB
 devid1 size 36.38TiB used 28.43TiB path
/dev/mapper/stripe0-vmbackup

# mount | grep vmbackup
/dev/mapper/stripe0-vmbackup on /vmbackup type btrfs
(rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,clear_cache,commit=300,subvolid=5,subvol=/)

dmesg is empty.

if i umount the volume i get traces (i already did a clear_cache 4 days
ago to recalculate the space_tree):

[545031.675797] [ cut here ]
[545031.725166] WARNING: CPU: 1 PID: 17711 at
fs/btrfs/extent-tree.c:5710 btrfs_free_block_groups+0x35a/0x400 [btrfs]()
[545031.778329] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandler
button btrfs dm_mod raid1 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq md_mod ixgbe mdio sg sd_mod
ahci ptp libahci megaraid_sas pps_core
[545032.081037] CPU: 1 PID: 17711 Comm: umount Tainted: G   O
4.4.15+43-ph #1
[545032.145078] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
02/18/2015
[545032.210238]   88010c40bcd8 bd3c712f

[545032.275650]  c03ec603 88010c40bd18 bd0837e7
0047a000
[545032.341525]   88105e0ea400 881054a76088
881054a76000
[545032.408500] Call Trace:
[545032.475272]  [] dump_stack+0x63/0x84
[545032.543620]  [] warn_slowpath_common+0x97/0xe0
[545032.612900]  [] warn_slowpath_null+0x1a/0x20
[545032.682026]  []
btrfs_free_block_groups+0x35a/0x400 [btrfs]
[545032.750297]  [] close_ctree+0x15b/0x330 [btrfs]
[545032.817085]  [] btrfs_put_super+0x19/0x20 [btrfs]
[545032.883439]  [] generic_shutdown_super+0x6f/0x100
[545032.949302]  [] kill_anon_super+0x16/0x30
[545033.014327]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
[545033.079031]  [] deactivate_locked_super+0x51/0x90
[545033.143275]  [] deactivate_super+0x4e/0x70
[545033.206535]  [] cleanup_mnt+0x43/0x90
[545033.268842]  [] __cleanup_mnt+0x12/0x20
[545033.331629]  [] task_work_run+0x7e/0xa0
[545033.393350]  [] exit_to_usermode_loop+0x66/0x95
[545033.454685]  [] syscall_return_slowpath+0xa6/0xf0
[545033.515485]  [] int_ret_from_sys_call+0x25/0x8f
[545033.575890] ---[ end trace bd985b05cc90617c ]---
[545033.636708] [ cut here ]
[545033.696339] WARNING: CPU: 1 PID: 17711 at
fs/btrfs/extent-tree.c:5711 btrfs_free_block_groups+0x3f4/0x400 [btrfs]()
[545033.758031] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandler
button btrfs dm_mod raid1 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq md_mod ixgbe mdio sg sd_mod
ahci ptp libahci megaraid_sas pps_core
[545034.095188] CPU: 1 PID: 17711 Comm: umount Tainted: GW  O
4.4.15+43-ph #1
[545034.166070] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
02/18/2015
[545034.236259]   88010c40bcd8 bd3c712f

[545034.307690]  c03ec603 88010c40bd18 bd0837e7
0047a000
[545034.379596]   88105e0ea400 881054a76088
881054a76000
[545034.452542] Call Trace:
[545034.525286]  [] dump_stack+0x63/0x84
[545034.599643]  [] warn_slowpath_common+0x97/0xe0
[545034.674894]  [] warn_slowpath_null+0x1a/0x20
[545034.750338]  []
btrfs_free_block_groups+0x3f4/0x400 [btrfs]
[545034.826354]  [] close_ctree+0x15b/0x330 [btrfs]
[545034.900758]  [] btrfs_put_super+0x19/0x20 [btrfs]
[545034.973612

Re: ENOSPC / no space on very large devices

2016-07-20 Thread Holger Hoffstätte
On 07/20/16 07:31, Stefan Priebe - Profihost AG wrote:
> Hi list,
> 
> while i didn't had the problem for some month i'm now getting ENOSPC on
> a regular basis on one host.

Well, it's getting better. :)

> if i umount the volume i get traces (i already did a clear_cache 4 days
> ago to recalculate the space_tree):
> 
> [545031.675797] [ cut here ]
> [545031.725166] WARNING: CPU: 1 PID: 17711 at
> fs/btrfs/extent-tree.c:5710 btrfs_free_block_groups+0x35a/0x400 [btrfs]()

This is "only" a warning, but as we can see below it indicates a real
problem. The warning was added only recently to for-next by the patch called
"Btrfs: warn_on for unaccounted spaces" [1], but I've had it in my tree
forever. Never seen the warning myself.

(snip)
> [545037.909700] BTRFS: space_info 4 has 18446743523026157568 free, is
> not full

Wow, ~18.4 exabytes really is a lot of free space. :)
So it looks like something underflowed the space_info and now things are
confused for about ~550 GB. Unfortunately I have no good idea how to fix
that. :(

> The kernel is something special - i'm using this one from holger:
> https://github.com/hhoffstaette/kernel-patches
> 
> which is basically a 4.4.15 + several patches especially a lot of btrfs
> patches up to 4.8 i think.

More like for-next with all the pagesize/sectorsize stuff carefully
avoided. I'm really looking forward to 4.8, this is becoming unwieldy..

-h

[1] 
https://git.kernel.org/cgit/linux/kernel/git/kdave/linux.git/commit/?h=for-next&id=d555b6c380c644af63dbdaa7cc14bba041a4e4dd

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: ENOSPC / no space on very large devices

2016-07-28 Thread Stefan Priebe - Profihost AG
here we go...

Am 20.07.2016 um 08:31 schrieb Wang Xiaoguang:
> hello,
> 
> On 07/20/2016 01:31 PM, Stefan Priebe - Profihost AG wrote:
>> Hi list,
>>
>> while i didn't had the problem for some month i'm now getting ENOSPC on
>> a regular basis on one host.
>>
>> It would be great if someone can help me debugging this.
>>
>> Some basic informations:
>> # touch /vmbackup/abc
>> touch: cannot touch `/vmbackup/abc': No space left on device
> When touch operation failed, would you please change dir to
> /sys/fs/btrfs/UUID/allocation/data/ and show me these files' value.
> And also files in /sys/fs/btrfs/UUID/allocation/metadata. thanks.
> Here UUID is your real uuid :)

/sys/fs/btrfs/ebcb9a5e-d784-4e17-9cd0-bc67fe7b1ed6/allocation/data]#
grep -H '' *
bytes_may_use:0
bytes_pinned:0
bytes_reserved:0
bytes_used:6175380234240
disk_total:6641093181440
disk_used:6175380234240
flags:1
grep: single: Is a directory
total_bytes:6641093181440
total_bytes_pinned:726104035328

 /sys/fs/btrfs/ebcb9a5e-d784-4e17-9cd0-bc67fe7b1ed6/allocation/metadata]# grep 
-H '' *
bytes_may_use:2089625649152
bytes_pinned:0
bytes_reserved:0
bytes_used:36823187456
disk_total:95563022336
disk_used:73646374912
grep: dup: Is a directory
flags:4
total_bytes:47781511168
total_bytes_pinned:-16792829952

Greets,
Stefan

> 
> Regards,
> Xiaoguang Wang
> 
>> # df -h /vmbackup/
>> FilesystemSize  Used Avail Use% Mounted on
>> /dev/mapper/stripe0-vmbackup   37T   28T  8,5T  77% /vmbackup
>>
>> # btrfs filesystem df /vmbackup/
>> Data, single: total=27.87TiB, used=27.39TiB
>> System, DUP: total=8.00MiB, used=4.34MiB
>> Metadata, DUP: total=286.50GiB, used=199.91GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs filesystem show /vmbackup/
>> Label: none  uuid: c8c3abf7-8280-4baa-bb51-a8c599e48002
>>  Total devices 1 FS bytes used 27.59TiB
>>  devid1 size 36.38TiB used 28.43TiB path
>> /dev/mapper/stripe0-vmbackup
>>
>> # mount | grep vmbackup
>> /dev/mapper/stripe0-vmbackup on /vmbackup type btrfs
>> (rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,clear_cache,commit=300,subvolid=5,subvol=/)
>>
>>
>> dmesg is empty.
>>
>> if i umount the volume i get traces (i already did a clear_cache 4 days
>> ago to recalculate the space_tree):
>>
>> [545031.675797] [ cut here ]
>> [545031.725166] WARNING: CPU: 1 PID: 17711 at
>> fs/btrfs/extent-tree.c:5710 btrfs_free_block_groups+0x35a/0x400 [btrfs]()
>> [545031.778329] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
>> mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
>> ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
>> ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
>> i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandler
>> button btrfs dm_mod raid1 raid456 async_raid6_recov async_memcpy
>> async_pq async_xor async_tx xor raid6_pq md_mod ixgbe mdio sg sd_mod
>> ahci ptp libahci megaraid_sas pps_core
>> [545032.081037] CPU: 1 PID: 17711 Comm: umount Tainted: G   O
>> 4.4.15+43-ph #1
>> [545032.145078] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
>> 02/18/2015
>> [545032.210238]   88010c40bcd8 bd3c712f
>> 
>> [545032.275650]  c03ec603 88010c40bd18 bd0837e7
>> 0047a000
>> [545032.341525]   88105e0ea400 881054a76088
>> 881054a76000
>> [545032.408500] Call Trace:
>> [545032.475272]  [] dump_stack+0x63/0x84
>> [545032.543620]  [] warn_slowpath_common+0x97/0xe0
>> [545032.612900]  [] warn_slowpath_null+0x1a/0x20
>> [545032.682026]  []
>> btrfs_free_block_groups+0x35a/0x400 [btrfs]
>> [545032.750297]  [] close_ctree+0x15b/0x330 [btrfs]
>> [545032.817085]  [] btrfs_put_super+0x19/0x20 [btrfs]
>> [545032.883439]  [] generic_shutdown_super+0x6f/0x100
>> [545032.949302]  [] kill_anon_super+0x16/0x30
>> [545033.014327]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
>> [545033.079031]  [] deactivate_locked_super+0x51/0x90
>> [545033.143275]  [] deactivate_super+0x4e/0x70
>> [545033.206535]  [] cleanup_mnt+0x43/0x90
>> [545033.268842]  [] __cleanup_mnt+0x12/0x20
>> [545033.331629]  [] task_work_run+0x7e/0xa0
>> [545033.393350]  [] exit_to_usermode_loop+0x66/0x95
>> [545033.454685]  [] syscall_return_slowpath+0xa6/0xf0
>> [545033.515485]  [] int_ret_from_sys_call+0x25/0x8f
>> [545033.575890] ---[ end trace bd985b05cc90617c ]---
>> [545033.636708] [ cut here ]
>> [545033.696339] WARNING: CPU: 1 PID: 17711 at
>> fs/btrfs/extent-tree.c:5711 btrfs_free_block_groups+0x3f4/0x400 [btrfs]()
>> [545033.758031] Modules linked in: netconsole ipt_REJECT nf_reject_ipv4
>> mpt3sas raid_class scsi_transport_sas xt_multiport iptable_filter
>> ip_tables x_tables 8021q garp bonding coretemp loop i40e(O) vxlan
>> ip6_udp_tunnel usbhid udp_tunnel sb_edac ehci_pci edac_core ehci_hcd
>> i2c_i801 i2c_core usbcore shpchp usb_common ipmi_si ipmi_msghandle

Re: ENOSPC / no space on very large devices

2016-07-28 Thread Stefan Priebe - Profihost AG

Am 20.07.2016 um 09:35 schrieb Holger Hoffstätte:
> On 07/20/16 07:31, Stefan Priebe - Profihost AG wrote:
>> Hi list,
>>
>> while i didn't had the problem for some month i'm now getting ENOSPC on
>> a regular basis on one host.
> 
> Well, it's getting better. :)

Again the same problem.

> 
>> if i umount the volume i get traces (i already did a clear_cache 4 days
>> ago to recalculate the space_tree):
>>
>> [545031.675797] [ cut here ]
>> [545031.725166] WARNING: CPU: 1 PID: 17711 at
>> fs/btrfs/extent-tree.c:5710 btrfs_free_block_groups+0x35a/0x400 [btrfs]()
> 
> This is "only" a warning, but as we can see below it indicates a real
> problem. The warning was added only recently to for-next by the patch called
> "Btrfs: warn_on for unaccounted spaces" [1], but I've had it in my tree
> forever. Never seen the warning myself.
> 
> (snip)
>> [545037.909700] BTRFS: space_info 4 has 18446743523026157568 free, is
>> not full
> 
> Wow, ~18.4 exabytes really is a lot of free space. :)
> So it looks like something underflowed the space_info and now things are
> confused for about ~550 GB. Unfortunately I have no good idea how to fix
> that. :(

umount triggered this one:
[983102.838217] [ cut here ]
[983102.864383] WARNING: CPU: 1 PID: 483 at fs/btrfs/extent-tree.c:5710
btrfs_free_block_groups+0x35a/0x400 [btrfs]()
[983102.894424] Modules linked in: netconsole xt_multiport
iptable_filter ip_tables x_tables 8021q garp bonding usbhid coretemp
loop xhci_pci ehci_pci xhci_hcd ehci_hcd i40e(O) sb_edac vxlan
ip6_udp_tunnel usbcore ipmi_si i2c_i801 shpchp usb_common udp_tunnel
edac_core ipmi_msghandler button btrfs dm_mod raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod igb
i2c_algo_bit i2c_core sg sd_mod ptp ahci libahci pps_core aacraid
[983103.043010] CPU: 1 PID: 483 Comm: umount Tainted: G   O
4.4.15+43-ph #1
[983103.084441] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
2.0 12/17/2015
[983103.127289]   880074673cd8 ad3c712f

[983103.171816]  c0298603 880074673d18 ad0837e7
000b2000
[983103.216766]   88103bd7e600 881037c92088
881037c92000
[983103.262776] Call Trace:
[983103.308602]  [] dump_stack+0x63/0x84
[983103.355609]  [] warn_slowpath_common+0x97/0xe0
[983103.403528]  [] warn_slowpath_null+0x1a/0x20
[983103.451297]  []
btrfs_free_block_groups+0x35a/0x400 [btrfs]
[983103.500439]  [] close_ctree+0x15b/0x330 [btrfs]
[983103.548805]  [] btrfs_put_super+0x19/0x20 [btrfs]
[983103.597122]  [] generic_shutdown_super+0x6f/0x100
[983103.645398]  [] kill_anon_super+0x16/0x30
[983103.693384]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
[983103.742430]  [] deactivate_locked_super+0x51/0x90
[983103.791501]  [] deactivate_super+0x4e/0x70
[983103.839979]  [] cleanup_mnt+0x43/0x90
[983103.889050]  [] __cleanup_mnt+0x12/0x20
[983103.937756]  [] task_work_run+0x7e/0xa0
[983103.986032]  [] exit_to_usermode_loop+0x66/0x95
[983104.035214]  [] syscall_return_slowpath+0xa6/0xf0
[983104.084312]  [] int_ret_from_sys_call+0x25/0x8f
[983104.134098] ---[ end trace ca97a745adcb888f ]---
[983104.184540] [ cut here ]
[983104.235514] WARNING: CPU: 1 PID: 483 at fs/btrfs/extent-tree.c:5711
btrfs_free_block_groups+0x3f4/0x400 [btrfs]()
[983104.290282] Modules linked in: netconsole xt_multiport
iptable_filter ip_tables x_tables 8021q garp bonding usbhid coretemp
loop xhci_pci ehci_pci xhci_hcd ehci_hcd i40e(O) sb_edac vxlan
ip6_udp_tunnel usbcore ipmi_si i2c_i801 shpchp usb_common udp_tunnel
edac_core ipmi_msghandler button btrfs dm_mod raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod igb
i2c_algo_bit i2c_core sg sd_mod ptp ahci libahci pps_core aacraid
[983104.536076] CPU: 1 PID: 483 Comm: umount Tainted: GW  O
4.4.15+43-ph #1
[983104.601962] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
2.0 12/17/2015
[983104.669312]   880074673cd8 ad3c712f

[983104.738337]  c0298603 880074673d18 ad0837e7
000b2000
[983104.807874]   88103bd7e600 881037c92088
881037c92000
[983104.878415] Call Trace:
[983104.948803]  [] dump_stack+0x63/0x84
[983105.020781]  [] warn_slowpath_common+0x97/0xe0
[983105.093734]  [] warn_slowpath_null+0x1a/0x20
[983105.166888]  []
btrfs_free_block_groups+0x3f4/0x400 [btrfs]
[983105.241461]  [] close_ctree+0x15b/0x330 [btrfs]
[983105.316744]  [] btrfs_put_super+0x19/0x20 [btrfs]
[983105.392021]  [] generic_shutdown_super+0x6f/0x100
[983105.465852]  [] kill_anon_super+0x16/0x30
[983105.537829]  [] btrfs_kill_super+0x1a/0xb0 [btrfs]
[983105.608078]  [] deactivate_locked_super+0x51/0x90
[983105.676494]  [] deactivate_super+0x4e/0x70
[983105.742614]  [] cleanup_mnt+0x43/0x90
[983105.807432]  [] __cleanup_mnt+0x12/0x20
[983105.871301]  [] task_work_run+0x7e/0xa0
[9