Re: Please review and comment, dealing with btrfs full issues
On Tue, May 06, 2014 at 05:43:24PM +0100, Hugo Mills wrote:
So in my case when I hit that case, I had to use dusage=0 to recover.
Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance
wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it
didn't need a destination for the data. That is actually an interesting
workaround for that case.
I've actually looked into implementing a smallest=n filter that
would taken only the n least-full chunks (by fraction) and balance
those. However, it's not entirely trivial to do efficiently with the
current filtering code.
I've prototyped something similar, to limit the number of balanced
chunks by a number. To achieve n least-full chunks would be an
iterative process of increasing the usage filter and limiting the number
of chunks until the desired N is reached.
N=n
F=0
while (N 0) {
balance -dusage=F,limit=N
N -= number of balanced chunks
F++
}
The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from
userspace.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)
On Wed, May 07, 2014 at 04:09:27PM +0200, David Sterba wrote:
On Tue, May 06, 2014 at 05:43:24PM +0100, Hugo Mills wrote:
So in my case when I hit that case, I had to use dusage=0 to recover.
Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance
wasn't empty - and it had nowhere to put it. Then when you did dusage=0,
it
didn't need a destination for the data. That is actually an interesting
workaround for that case.
I've actually looked into implementing a smallest=n filter that
would taken only the n least-full chunks (by fraction) and balance
those. However, it's not entirely trivial to do efficiently with the
current filtering code.
I've prototyped something similar, to limit the number of balanced
chunks by a number. To achieve n least-full chunks would be an
iterative process of increasing the usage filter and limiting the number
of chunks until the desired N is reached.
N=n
F=0
while (N 0) {
balance -dusage=F,limit=N
N -= number of balanced chunks
F++
}
The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from
userspace.
This is inefficient, because we've got an O(m) pass through all the
chunks for every call. If we reduce the number of calls by increasing
the increment of F (F+=3, say), then we risk overbalancing, or missing
out on smaller chunks we could have balanced earlier. From a practical
point of view, it may make little difference, but the computer
scientist in me is going ew.
The other method, for small n only, would be to construct the list
first, an O(m log n) operation for a filesystem of size m, requiring
O(n) storage, and then iterate over just those chunks. The problem
with that is the storage requirements, and keeping track of the state
of the list for restart purposes. [actually, there's probably an O(m)
algorithm to get the n smallest items, but those are a bit
complicated]
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- A diverse working environment: Di longer you vork here, di ---
verse it gets.
signature.asc
Description: Digital signature
Re: Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)
On Wed, May 07, 2014 at 03:23:01PM +0100, Hugo Mills wrote:
N=n
F=0
while (N 0) {
balance -dusage=F,limit=N
N -= number of balanced chunks
F++
}
The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from
userspace.
This is inefficient, because we've got an O(m) pass through all the
chunks for every call. If we reduce the number of calls by increasing
the increment of F (F+=3, say), then we risk overbalancing, or missing
out on smaller chunks we could have balanced earlier. From a practical
point of view, it may make little difference, but the computer
scientist in me is going ew.
I'm trying to find the practical way, no doubts about the inefficiencies.
The +1 increment was meant to outline the idea, I'm usually using the
sequence 0, 1, 5, 10, [etc +10].
I think we can afford some inaccuracy, I as a user would not mind if
there's some overbalancing (within a sane margin).
The other method, for small n only, would be to construct the list
first, an O(m log n) operation for a filesystem of size m, requiring
O(n) storage, and then iterate over just those chunks.
The size of filesystem matters, but the scanning phase of balance uses
in-memory structures and this should not be that bad for terabyte-sized
filesystems (ie. number of blockgoups will be some thousands).
Possibly we can stop looking for new chunks in the first phase of
balance when there are already N candidate chunks found, and process them.
The problem with that is the storage requirements, and keeping track
of the state of the list for restart purposes. [actually, there's
probably an O(m) algorithm to get the n smallest items, but those are
a bit complicated]
If the filesystem is under load, the chunks' usage may increase or
decrease in time and as we know, balance takes time, so the
chunk-todo-list may look different when next one is about to be
processed.
But yeah, this could be a cheaper check to skip a given chunk if it's
out of the filter criteria than going through the whole list again.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote: In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full: legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1 -dusage=50 will balance all chunks that are 50% *or less* used, Sorry, I actually meant to write 55 there. not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :) Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55? In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use. So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work. On Mon, May 05, 2014 at 07:09:22PM +0200, Brendan Hide wrote: Forgot this part: Also in your last example, you used -dusage=0 and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;) Correct. That FS was very mis-balanced. On Mon, May 05, 2014 at 02:36:09PM -0400, Calvin Walton wrote: The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes. I just added that tip, thank you. On Tue, May 06, 2014 at 02:41:16PM +1000, Russell Coker wrote: Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too. Good point. I added that tip too. Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
Hi, Marc. Inline below. :) On 2014/05/06 02:19 PM, Marc MERLIN wrote: On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote: In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full: legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1 -dusage=50 will balance all chunks that are 50% *or less* used, Sorry, I actually meant to write 55 there. not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :) Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55? As usual, it depends on what end-result you want. Paranoid rebalancing - always ensuring there are as many free chunks as possible - is totally unnecessary. There may be more good reasons to rebalance - but I'm only aware of two: a) to avoid ENOSPC due to running out of free chunks; and b) to change allocation type. If you want all chunks either full or empty (except for that last chunk which will be somewhere inbetween), -dusage=55 will get you 99% there. In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use. So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work. I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case. On Mon, May 05, 2014 at 07:09:22PM +0200, Brendan Hide wrote: Forgot this part: Also in your last example, you used -dusage=0 and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;) Correct. That FS was very mis-balanced. On Mon, May 05, 2014 at 02:36:09PM -0400, Calvin Walton wrote: The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes. I just added that tip, thank you. On Tue, May 06, 2014 at 02:41:16PM +1000, Russell Coker wrote: Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too. Good point. I added that tip too. Thanks, Marc -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On Tue, May 06, 2014 at 06:30:31PM +0200, Brendan Hide wrote: Hi, Marc. Inline below. :) On 2014/05/06 02:19 PM, Marc MERLIN wrote: On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote: In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full: legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1 -dusage=50 will balance all chunks that are 50% *or less* used, Sorry, I actually meant to write 55 there. not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :) Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55? As usual, it depends on what end-result you want. Paranoid rebalancing - always ensuring there are as many free chunks as possible - is totally unnecessary. There may be more good reasons to rebalance - but I'm only aware of two: a) to avoid ENOSPC due to running out of free chunks; and b) to change allocation type. c) its original reason: to redistribute the data on the FS, for example in the case of a new device being added or removed. If you want all chunks either full or empty (except for that last chunk which will be somewhere inbetween), -dusage=55 will get you 99% there. In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use. So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work. I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case. I've actually looked into implementing a smallest=n filter that would taken only the n least-full chunks (by fraction) and balance those. However, it's not entirely trivial to do efficiently with the current filtering code. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Hail and greetings. We are a flat-pack invasion force from --- Planet Ikea. We come in pieces. signature.asc Description: Digital signature
Re: Please review and comment, dealing with btrfs full issues
Brendan Hide posted on Tue, 06 May 2014 18:30:31 +0200 as excerpted: So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work. I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case. I've actually used -Xusage=0 (where X=m or d, obviously) for exactly that. If every last bit of filesystem is allocated so another chunk simply cannot be written in ordered to rewrite partially used chunks into, BUT the spread between allocated and actually used is quite high, there's a reasonably good chance that at least one of those allocated chunks is entirely empty, and -Xusage=0 allows returning it to the unallocated pool without actually requiring a new chunk allocation to do so. With luck, that will free at least one zero-usage chunk (two for metadata dup, but it would both allocate and return to unallocated in pairs as so it balances out), allowing the user to rerun balance, this time with a higher -Xusage=. The other known valid use-case for -Xusage=0 is when freeing the extraneous zero-usage single-mode chunks first created by mkfs.btrfs as part of the mkfs process, so they don't clutter up the btrfs filesystem df output. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Please review and comment, dealing with btrfs full issues
I've just written this new page: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html First, are there problems in it? Second, are there other FS full issues I should mention in it? Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On 05/05/14 14:16, Marc MERLIN wrote: I've just written this new page: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html First, are there problems in it? Second, are there other FS full issues I should mention in it? Thanks, Marc In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full: legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1 -dusage=50 will balance all chunks that are 50% *or less* used, not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :) In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use. The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that. -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On 05/05/14 19:07, Brendan Hide wrote: On 05/05/14 14:16, Marc MERLIN wrote: I've just written this new page: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html First, are there problems in it? Second, are there other FS full issues I should mention in it? Thanks, Marc In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full: legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1 -dusage=50 will balance all chunks that are 50% *or less* used, not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :) In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use. The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that. Forgot this part: Also in your last example, you used -dusage=0 and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;) -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On Mon, 2014-05-05 at 19:07 +0200, Brendan Hide wrote: On 05/05/14 14:16, Marc MERLIN wrote: I've just written this new page: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html First, are there problems in it? Second, are there other FS full issues I should mention in it? The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that. The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes. -- Calvin Walton calvin.wal...@kepstin.ca -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Please review and comment, dealing with btrfs full issues
On Mon, 5 May 2014 14:36:09 Calvin Walton wrote: The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes. I once had a situation where adding a new device failed due to lack of space. That was after I had tried to balance without adding a new device and to remove snapshots. http://etbe.coker.com.au/2014/04/26/btrfs-status-april-2014/ Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
