Re: Metadata balance fails ENOSPC

2016-12-05 Thread Duncan 
Stefan Priebe - Profihost AG posted on Mon, 05 Dec 2016 12:12:12 +0100 as
excerpted:

> isn't there a way to move free space to unallocated space again?

Yes, btrfs balance, but...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-05 Thread Stefan Priebe - Profihost AG 
isn't there a way to move free space to unallocated space again?


Am 03.12.2016 um 05:43 schrieb Andrei Borzenkov:
> 01.12.2016 18:48, Chris Murphy пишет:
>> On Thu, Dec 1, 2016 at 7:10 AM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>> Am 01.12.2016 um 14:51 schrieb Hans van Kranenburg:
 On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>  wrote:
> ...
>>
>> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
>> which does the same.
>>
>>
 # btrfs filesystem show /ssddisk/
 Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
 Total devices 1 FS bytes used 305.67GiB
 devid1 size 500.00GiB used 500.00GiB path /dev/vdb1

 # btrfs filesystem usage /ssddisk/
 Overall:
 Device size: 500.00GiB
 Device allocated:500.00GiB
 Device unallocated:1.05MiB
>>>
>>> Drive is actually fully allocated so if Btrfs needs to create a new
>>> chunk right now, it can't. However,
>>
>> Yes but there's lot of free space:
>> Free (estimated):193.46GiB  (min: 193.46GiB)
>>
>> How does this match?
>>
>>
>>> All three chunk types have quite a bit of unused space in them, so
>>> it's unclear why there's a no space left error.
>>>
>
> I remember discussion that balance always tries to pre-allocate one
> chunk in advance, and I believe there was patch to correct it but I am
> not sure whether it was merged.

 http://www.spinics.net/lists/linux-btrfs/msg56772.html
>>>
>>> Thanks - still don't understand why that one is not upstream or why it
>>> was reverted. Looks absolutely reasonable to me.
>>
>> It is upstream and hasn't been reverted.
>>
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/btrfs/volumes.c?id=refs/tags/v4.8.11
>> line 3650
>>
>> I would try Duncan's idea of using just one filter and seeing what happens:
>>
>> 'btrfs balance start -dusage=1 '
>>
> 
> Actually I just hit exactly the same symptoms on my VM where device was
> fully allocated and metadata balance failed, but data balance succeeded
> to free up space which allowed metadata balance to run too. This is
> under 4.8.10.
> 
> So it appears that balance logic between data and metadata is somehow
> different.
> 
> As this VM gets in 100% allocated condition fairly often I'd try to get
> better understanding next time.
> 
> 
>>
>> With enospc debug it says:
>> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
>> chunk for block group 839941881856
>> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
>>
>> It might be nice if this stated what kind of chunk it's trying to allocate.
>>
>>
>>
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-02 Thread Andrei Borzenkov 
01.12.2016 18:48, Chris Murphy пишет:
> On Thu, Dec 1, 2016 at 7:10 AM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> Am 01.12.2016 um 14:51 schrieb Hans van Kranenburg:
>>> On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
 On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
  wrote:
 ...
>
> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
> which does the same.
>
>
>>> # btrfs filesystem show /ssddisk/
>>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>>> Total devices 1 FS bytes used 305.67GiB
>>> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>>>
>>> # btrfs filesystem usage /ssddisk/
>>> Overall:
>>> Device size: 500.00GiB
>>> Device allocated:500.00GiB
>>> Device unallocated:1.05MiB
>>
>> Drive is actually fully allocated so if Btrfs needs to create a new
>> chunk right now, it can't. However,
>
> Yes but there's lot of free space:
> Free (estimated):193.46GiB  (min: 193.46GiB)
>
> How does this match?
>
>
>> All three chunk types have quite a bit of unused space in them, so
>> it's unclear why there's a no space left error.
>>

 I remember discussion that balance always tries to pre-allocate one
 chunk in advance, and I believe there was patch to correct it but I am
 not sure whether it was merged.
>>>
>>> http://www.spinics.net/lists/linux-btrfs/msg56772.html
>>
>> Thanks - still don't understand why that one is not upstream or why it
>> was reverted. Looks absolutely reasonable to me.
> 
> It is upstream and hasn't been reverted.
> 
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/btrfs/volumes.c?id=refs/tags/v4.8.11
> line 3650
> 
> I would try Duncan's idea of using just one filter and seeing what happens:
> 
> 'btrfs balance start -dusage=1 '
> 

Actually I just hit exactly the same symptoms on my VM where device was
fully allocated and metadata balance failed, but data balance succeeded
to free up space which allowed metadata balance to run too. This is
under 4.8.10.

So it appears that balance logic between data and metadata is somehow
different.

As this VM gets in 100% allocated condition fairly often I'd try to get
better understanding next time.


> 
> With enospc debug it says:
> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
> chunk for block group 839941881856
> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
> 
> It might be nice if this stated what kind of chunk it's trying to allocate.
> 
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Stefan Priebe - Profihost AG 

Am 01.12.2016 um 16:48 schrieb Chris Murphy:
> On Thu, Dec 1, 2016 at 7:10 AM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> Am 01.12.2016 um 14:51 schrieb Hans van Kranenburg:
>>> On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
 On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
  wrote:
 ...
>
> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
> which does the same.
>
>
>>> # btrfs filesystem show /ssddisk/
>>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>>> Total devices 1 FS bytes used 305.67GiB
>>> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>>>
>>> # btrfs filesystem usage /ssddisk/
>>> Overall:
>>> Device size: 500.00GiB
>>> Device allocated:500.00GiB
>>> Device unallocated:1.05MiB
>>
>> Drive is actually fully allocated so if Btrfs needs to create a new
>> chunk right now, it can't. However,
>
> Yes but there's lot of free space:
> Free (estimated):193.46GiB  (min: 193.46GiB)
>
> How does this match?
>
>
>> All three chunk types have quite a bit of unused space in them, so
>> it's unclear why there's a no space left error.
>>

 I remember discussion that balance always tries to pre-allocate one
 chunk in advance, and I believe there was patch to correct it but I am
 not sure whether it was merged.
>>>
>>> http://www.spinics.net/lists/linux-btrfs/msg56772.html
>>
>> Thanks - still don't understand why that one is not upstream or why it
>> was reverted. Looks absolutely reasonable to me.
> 
> It is upstream and hasn't been reverted.
> 
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/btrfs/volumes.c?id=refs/tags/v4.8.11
> line 3650
> 
> I would try Duncan's idea of using just one filter and seeing what happens:
> 
> 'btrfs balance start -dusage=1 '

see below:

[zabbix-db ~]# btrfs balance start -dusage=1 /ssddisk/
Done, had to relocate 0 out of 505 chunks
[zabbix-db ~]# btrfs balance start -dusage=10 /ssddisk/
Done, had to relocate 0 out of 505 chunks
[zabbix-db ~]# btrfs balance start -musage=1 /ssddisk/
ERROR: error during balancing '/ssddisk/': No space left on device
There may be more info in syslog - try dmesg | tail
[zabbix-db ~]# dmesg
[78306.288834] BTRFS warning (device vdb1): no space to allocate a new
chunk for block group 839941881856
[78306.289197] BTRFS info (device vdb1): 1 enospc errors during balance

> 
> 
> With enospc debug it says:
> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
> chunk for block group 839941881856
> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
> 
> It might be nice if this stated what kind of chunk it's trying to allocate.
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Chris Murphy 
On Thu, Dec 1, 2016 at 7:10 AM, Stefan Priebe - Profihost AG
 wrote:
>
> Am 01.12.2016 um 14:51 schrieb Hans van Kranenburg:
>> On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
>>> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>>>  wrote:
>>> ...

 Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
 which does the same.


>> # btrfs filesystem show /ssddisk/
>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>> Total devices 1 FS bytes used 305.67GiB
>> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>>
>> # btrfs filesystem usage /ssddisk/
>> Overall:
>> Device size: 500.00GiB
>> Device allocated:500.00GiB
>> Device unallocated:1.05MiB
>
> Drive is actually fully allocated so if Btrfs needs to create a new
> chunk right now, it can't. However,

 Yes but there's lot of free space:
 Free (estimated):193.46GiB  (min: 193.46GiB)

 How does this match?


> All three chunk types have quite a bit of unused space in them, so
> it's unclear why there's a no space left error.
>
>>>
>>> I remember discussion that balance always tries to pre-allocate one
>>> chunk in advance, and I believe there was patch to correct it but I am
>>> not sure whether it was merged.
>>
>> http://www.spinics.net/lists/linux-btrfs/msg56772.html
>
> Thanks - still don't understand why that one is not upstream or why it
> was reverted. Looks absolutely reasonable to me.

It is upstream and hasn't been reverted.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/btrfs/volumes.c?id=refs/tags/v4.8.11
line 3650

I would try Duncan's idea of using just one filter and seeing what happens:

'btrfs balance start -dusage=1 '


 With enospc debug it says:
 [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
 chunk for block group 839941881856
 [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance

It might be nice if this stated what kind of chunk it's trying to allocate.



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Stefan Priebe - Profihost AG 

Am 01.12.2016 um 14:51 schrieb Hans van Kranenburg:
> On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
>> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>>  wrote:
>> ...
>>>
>>> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
>>> which does the same.
>>>
>>>
> # btrfs filesystem show /ssddisk/
> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
> Total devices 1 FS bytes used 305.67GiB
> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>
> # btrfs filesystem usage /ssddisk/
> Overall:
> Device size: 500.00GiB
> Device allocated:500.00GiB
> Device unallocated:1.05MiB

 Drive is actually fully allocated so if Btrfs needs to create a new
 chunk right now, it can't. However,
>>>
>>> Yes but there's lot of free space:
>>> Free (estimated):193.46GiB  (min: 193.46GiB)
>>>
>>> How does this match?
>>>
>>>
 All three chunk types have quite a bit of unused space in them, so
 it's unclear why there's a no space left error.

>>
>> I remember discussion that balance always tries to pre-allocate one
>> chunk in advance, and I believe there was patch to correct it but I am
>> not sure whether it was merged.
> 
> http://www.spinics.net/lists/linux-btrfs/msg56772.html

Thanks - still don't understand why that one is not upstream or why it
was reverted. Looks absolutely reasonable to me. Other option would be
to make it possible to make allocated unused space unallocted again - no
idea how todo that.

> 
 Try remounting with enoscp_debug, and then trigger the problem again,
 and post the resulting kernel messages.
>>>
>>> With enospc debug it says:
>>> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
>>> chunk for block group 839941881856
>>> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Hans van Kranenburg 
On 12/01/2016 09:12 AM, Andrei Borzenkov wrote:
> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>  wrote:
> ...
>>
>> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
>> which does the same.
>>
>>
 # btrfs filesystem show /ssddisk/
 Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
 Total devices 1 FS bytes used 305.67GiB
 devid1 size 500.00GiB used 500.00GiB path /dev/vdb1

 # btrfs filesystem usage /ssddisk/
 Overall:
 Device size: 500.00GiB
 Device allocated:500.00GiB
 Device unallocated:1.05MiB
>>>
>>> Drive is actually fully allocated so if Btrfs needs to create a new
>>> chunk right now, it can't. However,
>>
>> Yes but there's lot of free space:
>> Free (estimated):193.46GiB  (min: 193.46GiB)
>>
>> How does this match?
>>
>>
>>> All three chunk types have quite a bit of unused space in them, so
>>> it's unclear why there's a no space left error.
>>>
> 
> I remember discussion that balance always tries to pre-allocate one
> chunk in advance, and I believe there was patch to correct it but I am
> not sure whether it was merged.

http://www.spinics.net/lists/linux-btrfs/msg56772.html

>>> Try remounting with enoscp_debug, and then trigger the problem again,
>>> and post the resulting kernel messages.
>>
>> With enospc debug it says:
>> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
>> chunk for block group 839941881856
>> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread E V 
I've frequently seen free space cache corruption lead to phantom
ENOSPC. You could try clearing the space cache, and/or mounting with
nospache_cache.

On Thu, Dec 1, 2016 at 6:55 AM, Stefan Priebe - Profihost AG
 wrote:
>
> Am 01.12.2016 um 09:12 schrieb Andrei Borzenkov:
>> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>>  wrote:
>> ...
>>>
>>> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
>>> which does the same.
>>>
>>>
> # btrfs filesystem show /ssddisk/
> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
> Total devices 1 FS bytes used 305.67GiB
> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>
> # btrfs filesystem usage /ssddisk/
> Overall:
> Device size: 500.00GiB
> Device allocated:500.00GiB
> Device unallocated:1.05MiB

 Drive is actually fully allocated so if Btrfs needs to create a new
 chunk right now, it can't. However,
>>>
>>> Yes but there's lot of free space:
>>> Free (estimated):193.46GiB  (min: 193.46GiB)
>>>
>>> How does this match?
>>>
>>>
 All three chunk types have quite a bit of unused space in them, so
 it's unclear why there's a no space left error.

>>
>> I remember discussion that balance always tries to pre-allocate one
>> chunk in advance, and I believe there was patch to correct it but I am
>> not sure whether it was merged.
>
> Is there otherwise a possibility to make the free space unallocated again?
>
> Stefan
>
>>
 Try remounting with enoscp_debug, and then trigger the problem again,
 and post the resulting kernel messages.
>>>
>>> With enospc debug it says:
>>> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
>>> chunk for block group 839941881856
>>> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Stefan Priebe - Profihost AG 

Am 01.12.2016 um 09:12 schrieb Andrei Borzenkov:
> On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
>  wrote:
> ...
>>
>> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
>> which does the same.
>>
>>
 # btrfs filesystem show /ssddisk/
 Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
 Total devices 1 FS bytes used 305.67GiB
 devid1 size 500.00GiB used 500.00GiB path /dev/vdb1

 # btrfs filesystem usage /ssddisk/
 Overall:
 Device size: 500.00GiB
 Device allocated:500.00GiB
 Device unallocated:1.05MiB
>>>
>>> Drive is actually fully allocated so if Btrfs needs to create a new
>>> chunk right now, it can't. However,
>>
>> Yes but there's lot of free space:
>> Free (estimated):193.46GiB  (min: 193.46GiB)
>>
>> How does this match?
>>
>>
>>> All three chunk types have quite a bit of unused space in them, so
>>> it's unclear why there's a no space left error.
>>>
> 
> I remember discussion that balance always tries to pre-allocate one
> chunk in advance, and I believe there was patch to correct it but I am
> not sure whether it was merged.

Is there otherwise a possibility to make the free space unallocated again?

Stefan

> 
>>> Try remounting with enoscp_debug, and then trigger the problem again,
>>> and post the resulting kernel messages.
>>
>> With enospc debug it says:
>> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
>> chunk for block group 839941881856
>> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Andrei Borzenkov 
On Thu, Dec 1, 2016 at 10:49 AM, Stefan Priebe - Profihost AG
 wrote:
...
>
> Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
> which does the same.
>
>
>>> # btrfs filesystem show /ssddisk/
>>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>>> Total devices 1 FS bytes used 305.67GiB
>>> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>>>
>>> # btrfs filesystem usage /ssddisk/
>>> Overall:
>>> Device size: 500.00GiB
>>> Device allocated:500.00GiB
>>> Device unallocated:1.05MiB
>>
>> Drive is actually fully allocated so if Btrfs needs to create a new
>> chunk right now, it can't. However,
>
> Yes but there's lot of free space:
> Free (estimated):193.46GiB  (min: 193.46GiB)
>
> How does this match?
>
>
>> All three chunk types have quite a bit of unused space in them, so
>> it's unclear why there's a no space left error.
>>

I remember discussion that balance always tries to pre-allocate one
chunk in advance, and I believe there was patch to correct it but I am
not sure whether it was merged.

>> Try remounting with enoscp_debug, and then trigger the problem again,
>> and post the resulting kernel messages.
>
> With enospc debug it says:
> [39193.425682] BTRFS warning (device vdb1): no space to allocate a new
> chunk for block group 839941881856
> [39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-12-01 Thread Duncan 
Chris Murphy posted on Wed, 30 Nov 2016 16:02:29 -0700 as excerpted:

> On Wed, Nov 30, 2016 at 2:03 PM, Stefan Priebe - Profihost AG
>  wrote:
>> Hello,
>>
>> # btrfs balance start -v -dusage=0 -musage=1 /ssddisk/
>> Dumping filters: flags 0x7, state 0x0, force is off
>>   DATA (flags 0x2): balancing, usage=0
>>   METADATA (flags 0x2): balancing, usage=1
>>   SYSTEM (flags 0x2): balancing, usage=1
>> ERROR: error during balancing '/ssddisk/': No space left on device
>> There may be more info in syslog - try dmesg | tail
> 
> You haven't provided kernel messages at the time of the error.
> 
> Also useful is the kernel version.

I won't disagree here as often it's kernel-version-specific behavior in 
question, but in this case I think the behavior is generic and the 
question can thus be answered on that basis, without the kernel version 
or dmesg output.

@ Chris: Note that the ENOSPC wasn't during ordinary use, but 
/specifically/ during balance, which behaves a bit differently regarding 
ENOSPC, and I believe it's that version-generic behavior difference 
that's in focus, here.

>> # btrfs filesystem show /ssddisk/
>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>> Total devices 1 FS bytes used 305.67GiB
>  devid1 size 500.00GiB used 500.00GiB path /dev/vdb1

Device line says 100% used (meaning allocated).  The below simply shows 
it a different way, confirming the 100% used.

>> # btrfs filesystem usage /ssddisk/
>> Overall:
>> Device size: 500.00GiB
>> Device allocated:500.00GiB
>> Device unallocated:1.05MiB
> 
> Drive is actually fully allocated so if Btrfs needs to create a new
> chunk right now, it can't.

... And that right there is the problem.

When doing chunk consolidation, with one exception noted below, btrfs 
balance creates new chunks to write into, then rewrites the content from 
the old into the new.  But there's no space left (1 MiB isn't enough) 
unallocated to allocate new chunks from, so balance errors out with 
ENOSPC.

>> Data,single: Size:483.97GiB, Used:298.18GiB
>>/dev/vdb1 483.97GiB
>>
>> Metadata,single: Size:16.00GiB, Used:7.51GiB
>>/dev/vdb1  16.00GiB
>>
>> System,single: Size:32.00MiB, Used:144.00KiB
>>/dev/vdb1  32.00MiB
> 
> All three chunk types have quite a bit of unused space in them, so it's
> unclear why there's a no space left error.

Normal usage can still write into the existing chunks since they're not 
yet entirely full, but that's not where the error occurred.  There's no 
space left unallocated to allocate further chunks from, and that's what 
balance, with one single exception, must do first, allocate a new chunk 
in ordered to write into, so it errors out.

The one single exception is when there's actually nothing to rewrite, the 
usage=0 case, in which case balance will simply erase any entirely empty 
chunks of the appropriate type (-d=data, -m=metadata).

This _used_ to be required somewhat regularly, as the kernel knew how to 
allocate new chunks but couldn't deallocate chunks, even entirely empty 
chunks, without a balance.  However, since 3.16 (IIRC), the kernel has 
been able to deallocate entirely empty chunks entirely on its own 
(automatically), and does so reasonably regularly in normal usage, so the 
issue of zero-sized chunks is far rarer than it used to be.

But apparently there's still a bug or two somewhere, as we still get 
reports of the usage=0 filter actually deallocating some empty chunks 
back to unallocated, even on kernels that should be doing that 
automatically.  It's not as common as it once was, but it does still 
happen.

So the usage=0 filter, the only case where the kernel doesn't have to 
create a new chunk in ordered to clear space during a balance, because 
it's not actually writing a new chunk, only deleting an empty one, does 
still make sense to try, because sometimes it _does_ work, and in the 
100% allocated case it's the simplest thing to try so it's worth trying 
even tho there's a good chance it won't work, because the kernel is 
/supposed/ to be removing those chunks automatically now, and /usually/ 
does just that.


OK, so what was wrong with the above command, and what should be tried 
instead?

The above command used TWO filters, -dusage=0 -musage=1 .  It choked on 
the -musage=1, apparently because it tried a less-than 1% full but not /
entirely/ empty metadata chunk first, before trying data chunks with the 
-dusuage=0, which should have succeeded, even if it found no empty data 
chunks to remove.

So the fix is to try either -dusage=0 -musage=0 together, first, or to 
try -dusage by itself first (and possibly -musage=0 after that), before 
trying -musage=1.

If it works and there are empty chunks of either type that can be 
removed, hopefully that will free up enough space to write at least one 
more metadata chunk, leaving room to create at least the one more (it'd 
be two with

Re: Metadata balance fails ENOSPC

2016-11-30 Thread Stefan Priebe - Profihost AG 
Am 01.12.2016 um 00:02 schrieb Chris Murphy:
> On Wed, Nov 30, 2016 at 2:03 PM, Stefan Priebe - Profihost AG
>  wrote:
>> Hello,
>>
>> # btrfs balance start -v -dusage=0 -musage=1 /ssddisk/
>> Dumping filters: flags 0x7, state 0x0, force is off
>>   DATA (flags 0x2): balancing, usage=0
>>   METADATA (flags 0x2): balancing, usage=1
>>   SYSTEM (flags 0x2): balancing, usage=1
>> ERROR: error during balancing '/ssddisk/': No space left on device
>> There may be more info in syslog - try dmesg | tail
> 
> You haven't provided kernel messages at the time of the error.

Kernel Message:
[  429.107723] BTRFS info (device vdb1): 1 enospc errors during balance

> Also useful is the kernel version.

Custom 4.4 kernel with patches up to 4.10. But i already tried 4.9-rc7
which does the same.


>> # btrfs filesystem show /ssddisk/
>> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
>> Total devices 1 FS bytes used 305.67GiB
>> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>>
>> # btrfs filesystem usage /ssddisk/
>> Overall:
>> Device size: 500.00GiB
>> Device allocated:500.00GiB
>> Device unallocated:1.05MiB
> 
> Drive is actually fully allocated so if Btrfs needs to create a new
> chunk right now, it can't. However,

Yes but there's lot of free space:
Free (estimated):193.46GiB  (min: 193.46GiB)

How does this match?


> All three chunk types have quite a bit of unused space in them, so
> it's unclear why there's a no space left error.
> 
> Try remounting with enoscp_debug, and then trigger the problem again,
> and post the resulting kernel messages.

With enospc debug it says:
[39193.425682] BTRFS warning (device vdb1): no space to allocate a new
chunk for block group 839941881856
[39193.426033] BTRFS info (device vdb1): 1 enospc errors during balance

Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata balance fails ENOSPC

2016-11-30 Thread Chris Murphy 
On Wed, Nov 30, 2016 at 2:03 PM, Stefan Priebe - Profihost AG
 wrote:
> Hello,
>
> # btrfs balance start -v -dusage=0 -musage=1 /ssddisk/
> Dumping filters: flags 0x7, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=0
>   METADATA (flags 0x2): balancing, usage=1
>   SYSTEM (flags 0x2): balancing, usage=1
> ERROR: error during balancing '/ssddisk/': No space left on device
> There may be more info in syslog - try dmesg | tail

You haven't provided kernel messages at the time of the error.

Also useful is the kernel version.



>
> # btrfs filesystem show /ssddisk/
> Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
> Total devices 1 FS bytes used 305.67GiB
> devid1 size 500.00GiB used 500.00GiB path /dev/vdb1
>
> # btrfs filesystem usage /ssddisk/
> Overall:
> Device size: 500.00GiB
> Device allocated:500.00GiB
> Device unallocated:1.05MiB

Drive is actually fully allocated so if Btrfs needs to create a new
chunk right now, it can't. However,



>
> Data,single: Size:483.97GiB, Used:298.18GiB
>/dev/vdb1 483.97GiB
>
> Metadata,single: Size:16.00GiB, Used:7.51GiB
>/dev/vdb1  16.00GiB
>
> System,single: Size:32.00MiB, Used:144.00KiB
>/dev/vdb1  32.00MiB

All three chunk types have quite a bit of unused space in them, so
it's unclear why there's a no space left error.

Try remounting with enoscp_debug, and then trigger the problem again,
and post the resulting kernel messages.


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Metadata balance fails ENOSPC

2016-11-30 Thread Stefan Priebe - Profihost AG 
Hello,

# btrfs balance start -v -dusage=0 -musage=1 /ssddisk/
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
  METADATA (flags 0x2): balancing, usage=1
  SYSTEM (flags 0x2): balancing, usage=1
ERROR: error during balancing '/ssddisk/': No space left on device
There may be more info in syslog - try dmesg | tail

# btrfs filesystem show /ssddisk/
Label: none  uuid: a69d2e90-c2ca-4589-9876-234446868adc
Total devices 1 FS bytes used 305.67GiB
devid1 size 500.00GiB used 500.00GiB path /dev/vdb1

# btrfs filesystem usage /ssddisk/
Overall:
Device size: 500.00GiB
Device allocated:500.00GiB
Device unallocated:1.05MiB
Device missing:  0.00B
Used:305.69GiB
Free (estimated):185.78GiB  (min: 185.78GiB)
Data ratio:   1.00
Metadata ratio:   1.00
Global reserve:  512.00MiB  (used: 608.00KiB)

Data,single: Size:483.97GiB, Used:298.18GiB
   /dev/vdb1 483.97GiB

Metadata,single: Size:16.00GiB, Used:7.51GiB
   /dev/vdb1  16.00GiB

System,single: Size:32.00MiB, Used:144.00KiB
   /dev/vdb1  32.00MiB

Unallocated:
   /dev/vdb1   1.05MiB

How can i make it balancing again?

Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html