Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Hugo Mills]

On Fri, Nov 20, 2015 at 08:21:31AM -0500, Austin S Hemmelgarn wrote:
> On 2015-11-20 06:39, Dmitry Katsubo wrote:
> >If I may add:
> >
> >Information for "System"
> >
> >   System, DUP: total=32.00MiB, used=16.00KiB
> >
> >is also quite technical, as for end user system = metadata (one can call
> >it "filesystem metadata" perhaps). For simplicity the numbers can be
> >added to "Metadata" thus eliminating that line as well.
> >
> >For those power users who really want to see the tiny details like
> >"System" and "GlobalReserve" I suggest to implement "-v" flag:
> >
> ># btrfs fi usage -v
> Actually, I really like this idea, one of the questions I get asked
> when I show people BTRFS is the difference between System and
> Metadata, and it's not always easy to explain to somebody who
> doesn't have a background in filesystem development.  For some
> reason, people seem to have trouble with the concept that the system
> tree is an index of the other trees.

   Actually, it's not that in the system chunks. :)

   System chunks contain the chunk tree, not the tree of tree roots.
They're special (and small) because they're listed explicitly by devid
and physical offset at the end of the superblock, and allow the FS to
read them first so that it can bootstrap the logical:physical mapping
table before it starts reading all the other metadata like the tree of
tree roots (which is "normal" metadata).

   Hugo.

>  In general, it doesn't make
> sense for most non-debugging cases to have it listed separate from
> the Metadata (they always have the same profile unless you're part
> way through a conversion, and it really is metadata, just slightly
> higher level than the usual metadata chunks).
> >
> >On 2015-11-19 03:16, Duncan wrote:
> >>Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
> >>
> >>>Although the metadata output is showing that you still have about 512M
> >>>available, but the 512M is Global Reserved space, or the unknown one.
> >>
> >>Unknown here, as the userspace (btrfs-progs) is evidently too old to show
> >>it as global reserve, as it does in newer versions...
> >>
> >>>The output is really a little confusing. I'd like the change the output
> >>>by adding global reserved into metadata used space and make it a sub
> >>>item for metadata.
> >>
> >>Thanks for the clarification.  It's most helpful, here. =:^)
> >>
> >>I've at times wondered if global reserve folded into one of the other
> >>settings.  Apparently it comes from the metadata allocation, but while
> >>metadata is normally dup (single-device btrfs) or raid1 (multi-device),
> >>global reserve is single.
> >>
> >>It would have been nice if that sort of substructure was described a bit
> >>better when global reserve first made its appearance, at least in the
> >>patch descriptions and release announcement, if not then yet in btrfs fi
> >>df output, first implementations being what they are.  But regardless,
> >>now at least it should be clear for list regulars who read this thread
> >>anyway, since the above reasonably clarifies things.
> >>
> >>As for btrfs fi df, making global reserve a metadata subentry there would
> >>be one way to deal with it, preserving the exposure of the additional
> >>data provided by that line (here, the fact that global reserve is
> >>actually being used, underlining the fact that the filesystem is severely
> >>short on space).
> >>
> >>Another way of handling it would be to simply add the global reserve into
> >>the metadata used figure before printing it, eliminating the separate
> >>global reserve line, and changing the upthread posted metadata line from
> >>8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the
> >>case if the 512 MiB of global reserve indeed comes from the metadata
> >>allocation.  This would more clearly show how full metadata actually is
> >>without the added complexity of an additional global reserve line, but
> >>would lose the fact that global reserve is actually in use, that the
> >>broken out global reserve line exposes.
> >>
> >>I'd actually argue in favor of the latter, directly folding global
> >>reserve allocation into metadata used, since it'd both be simpler, and
> >>more consistent if for instance btrfs fi usage didn't report separate
> >>global reserve in the overall stats, but fail to report it in the per-
> >>device stats and in btrfs dev usage.
> >>
> >>Either way would make much clearer that metadata is actually running out
> >>than the current report layout does, since "metadata used" would then
> >>either explicitly or implicitly include the global reserve.
> >>
> >
> >
> 
> 



-- 
Hugo Mills | Startle, startle, little twink. How I wonder what
hugo@... carfax.org.uk | you think.
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Austin S Hemmelgarn]

On 2015-11-20 08:27, Hugo Mills wrote:

On Fri, Nov 20, 2015 at 08:21:31AM -0500, Austin S Hemmelgarn wrote:

On 2015-11-20 06:39, Dmitry Katsubo wrote:

If I may add:

Information for "System"

   System, DUP: total=32.00MiB, used=16.00KiB

is also quite technical, as for end user system = metadata (one can call
it "filesystem metadata" perhaps). For simplicity the numbers can be
added to "Metadata" thus eliminating that line as well.

For those power users who really want to see the tiny details like
"System" and "GlobalReserve" I suggest to implement "-v" flag:

# btrfs fi usage -v

Actually, I really like this idea, one of the questions I get asked
when I show people BTRFS is the difference between System and
Metadata, and it's not always easy to explain to somebody who
doesn't have a background in filesystem development.  For some
reason, people seem to have trouble with the concept that the system
tree is an index of the other trees.


Actually, it's not that in the system chunks. :)

System chunks contain the chunk tree, not the tree of tree roots.
They're special (and small) because they're listed explicitly by devid
and physical offset at the end of the superblock, and allow the FS to
read them first so that it can bootstrap the logical:physical mapping
table before it starts reading all the other metadata like the tree of
tree roots (which is "normal" metadata).

I guess my understanding was wrong then.  Thanks for the explanation.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Dmitry Katsubo]

Many thanks to Duncan for such a verbose clarification. I am thinking
about another parallel similar to SimSity, and that is memory management
in virtual machines like Java. If heap is full, it does not really mean
that there is no free memory. In this case JVM forces garbage collector
and if after that procedure no memory was released, then it signals this
to application by raising OutOfMemoryError.

I think the similar should happen in btrfs: ENOSPC is returned to
application only when there is really no space left. If no chunk can be
further allocated, btrfs should check all "deleted" data chunks and and
return them to unallocated pool. I would expect this automatic "cleanup"
from modern filesystem.

This behaviour not necessarily should be a default one, as one can argue
that:
* Such cleanup procedure may freeze the calling process for a
considerable time, as btrfs would need to walk all allocated chunks to
find candidates for release.
* Filesystem will perhaps anyway run of free space soon, so why not to
fallback with error earlier? (for example, one process is intensively
writing the log)

It would be nice to have the automatic "cleanup" function controlled by
some /sys/fs/btrfs/features variable, which if set to 1, forces btrfs to
do its best to allocate the chunk before giving up and returning ENOSPC,
sacrificing response time of the process/application.

On 2015-11-20 04:14, Duncan wrote:
> linux-btrfs.tebulin posted on Thu, 19 Nov 2015 18:56:45 + as
> excerpted:
> 
> Meta-comment:
> 
> Apparently that attribution should actually be to Hugo Mills.  I've no 
> idea what went wrong, but at least here as received from gmane.org, the 
> from header really does say linux-btrfs.tebulin, so something obviously 
> bugged out somewhere!
> 
> 
> Meanwhile discussing btrfs data/metadata allocation, vs. usage of that 
> allocation, Hugo also known here as tebulin, explained...
> 
>> If you've ever played SimCity, the allocation process is like zoning --
>> you say what kind of thing can go on the space, but it's not actually
>> used until something gets built on it.
> 
> Very nice analogy.  Thanks. =:^)
> 
> Tho I'd actually put it in terms of the real thing that sim city 
> simulates, thus eliminating the proprietary comparison.  A city zones an 
> area one way or another, restricting what can be built there -- you can't 
> put heavy industry in a residential zone.  But the lot is still empty 
> until something's actually built there.
> 
> And if the city has all its area zoned residential, and some company 
> wants to build a plant there (generally considered a good thing as it'll 
> provide employment), there's a process that allows rezoning.
> 
> In btrfs, there's four types of allocations aka "zones":
> 
> 1) Unallocated (unzoned)
> 
> Can be used for anything but must be allocated/zoned first
> 
> 2) System
> 
> Critical but limited in size and generally only allocated at mkfs or when 
> adding a new device.
> 
> 3) Data
> 
> The actual file storage, generally the largest allocation.
> 
> 4) Metadata
> 
> Information /about/ the files, where they are located (the location and 
> size of individual extents), ownership and permissions, date stamps, 
> checksums, and for very small files (a few KiB), sometimes the file 
> content itself.
> 
> 4a) Global reserve
> 
> A small part of metadata reserved for emergency use only.  Btrfs is 
> pretty strict about its use, and will generally error out with ENOSPC if 
> metadata space other than the global reserve is used up, before actually 
> using this global reserve.  As a result, any of the global reserve used 
> at all indicates a filesystem in very severe straits, crisis mode.
> 
> 
> As it happens, btrfs in practice tends to be a bit liberal about 
> allocating/zoning data chunks, since it's easier to find bigger blocks of 
> space in unallocated space than it is in busy partly used data space.  
> (Think of a big shopping center.  It's easier to build it in outlying 
> areas that haven't been built up yet, where many whole blocks worth of 
> space can be allocated/zoned at once, than it is in the city center, 
> where even finding a single whole block vacant, is near impossible.)
> 
> Over time, therefore, more and more space tends to be allocated to data, 
> while existing data space, like those blocks near city center, may have 
> most of its files/buildings deleted, but still have a couple still 
> occupied.
> 
> Btrfs balancing, then, is comparable to the city functions of 
> condemnation and rezoning to vacant/unallocated, forcing remaining 
> occupants, most commonly data zoned, to move out of the way so the area 
> can be reclaimed for other usage.  Then it can be rezoned to data again, 
> or to metadata, whatever needs it.
> 
> 
> (FWIW, I played the original sim city, but IIRC it wasn't sophisticated 
> enough to have zoning yet.  Of course I've not played anything recent as 
> to my knowledge it's not freedomware, and since I no longer agree 

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Austin S Hemmelgarn]

On 2015-11-20 06:39, Dmitry Katsubo wrote:

If I may add:

Information for "System"

   System, DUP: total=32.00MiB, used=16.00KiB

is also quite technical, as for end user system = metadata (one can call
it "filesystem metadata" perhaps). For simplicity the numbers can be
added to "Metadata" thus eliminating that line as well.

For those power users who really want to see the tiny details like
"System" and "GlobalReserve" I suggest to implement "-v" flag:

# btrfs fi usage -v
Actually, I really like this idea, one of the questions I get asked when 
I show people BTRFS is the difference between System and Metadata, and 
it's not always easy to explain to somebody who doesn't have a 
background in filesystem development.  For some reason, people seem to 
have trouble with the concept that the system tree is an index of the 
other trees.  In general, it doesn't make sense for most non-debugging 
cases to have it listed separate from the Metadata (they always have the 
same profile unless you're part way through a conversion, and it really 
is metadata, just slightly higher level than the usual metadata chunks).


On 2015-11-19 03:16, Duncan wrote:

Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:


Although the metadata output is showing that you still have about 512M
available, but the 512M is Global Reserved space, or the unknown one.


Unknown here, as the userspace (btrfs-progs) is evidently too old to show
it as global reserve, as it does in newer versions...


The output is really a little confusing. I'd like the change the output
by adding global reserved into metadata used space and make it a sub
item for metadata.


Thanks for the clarification.  It's most helpful, here. =:^)

I've at times wondered if global reserve folded into one of the other
settings.  Apparently it comes from the metadata allocation, but while
metadata is normally dup (single-device btrfs) or raid1 (multi-device),
global reserve is single.

It would have been nice if that sort of substructure was described a bit
better when global reserve first made its appearance, at least in the
patch descriptions and release announcement, if not then yet in btrfs fi
df output, first implementations being what they are.  But regardless,
now at least it should be clear for list regulars who read this thread
anyway, since the above reasonably clarifies things.

As for btrfs fi df, making global reserve a metadata subentry there would
be one way to deal with it, preserving the exposure of the additional
data provided by that line (here, the fact that global reserve is
actually being used, underlining the fact that the filesystem is severely
short on space).

Another way of handling it would be to simply add the global reserve into
the metadata used figure before printing it, eliminating the separate
global reserve line, and changing the upthread posted metadata line from
8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the
case if the 512 MiB of global reserve indeed comes from the metadata
allocation.  This would more clearly show how full metadata actually is
without the added complexity of an additional global reserve line, but
would lose the fact that global reserve is actually in use, that the
broken out global reserve line exposes.

I'd actually argue in favor of the latter, directly folding global
reserve allocation into metadata used, since it'd both be simpler, and
more consistent if for instance btrfs fi usage didn't report separate
global reserve in the overall stats, but fail to report it in the per-
device stats and in btrfs dev usage.

Either way would make much clearer that metadata is actually running out
than the current report layout does, since "metadata used" would then
either explicitly or implicitly include the global reserve.









smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[linux-btrfs . tebulin]

Am 20.11.2015 um 04:14 schrieb Duncan:
> Meta-comment:
> 
> Apparently that attribution should actually be to Hugo Mills.  I've no 
> idea what went wrong, but at least here as received from gmane.org, the 
> from header really does say linux-btrfs.tebulin, so something obviously 
> bugged out somewhere!

Meta-Answer: My fault

I accidentally exposed my spamgourmet proxy address to Hugo by taking
him CC. He replied to all assuming it would go directly to the list, but
the proxy address masked his original address an put id in it.

Sorry for messing up!

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Dmitry Katsubo]

If I may add:

Information for "System"

  System, DUP: total=32.00MiB, used=16.00KiB

is also quite technical, as for end user system = metadata (one can call
it "filesystem metadata" perhaps). For simplicity the numbers can be
added to "Metadata" thus eliminating that line as well.

For those power users who really want to see the tiny details like
"System" and "GlobalReserve" I suggest to implement "-v" flag:

# btrfs fi usage -v

On 2015-11-19 03:16, Duncan wrote:
> Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
> 
>> Although the metadata output is showing that you still have about 512M
>> available, but the 512M is Global Reserved space, or the unknown one.
> 
> Unknown here, as the userspace (btrfs-progs) is evidently too old to show 
> it as global reserve, as it does in newer versions...
> 
>> The output is really a little confusing. I'd like the change the output
>> by adding global reserved into metadata used space and make it a sub
>> item for metadata.
> 
> Thanks for the clarification.  It's most helpful, here. =:^)
> 
> I've at times wondered if global reserve folded into one of the other 
> settings.  Apparently it comes from the metadata allocation, but while 
> metadata is normally dup (single-device btrfs) or raid1 (multi-device), 
> global reserve is single.
> 
> It would have been nice if that sort of substructure was described a bit 
> better when global reserve first made its appearance, at least in the 
> patch descriptions and release announcement, if not then yet in btrfs fi 
> df output, first implementations being what they are.  But regardless, 
> now at least it should be clear for list regulars who read this thread 
> anyway, since the above reasonably clarifies things.
> 
> As for btrfs fi df, making global reserve a metadata subentry there would 
> be one way to deal with it, preserving the exposure of the additional 
> data provided by that line (here, the fact that global reserve is 
> actually being used, underlining the fact that the filesystem is severely 
> short on space).
> 
> Another way of handling it would be to simply add the global reserve into 
> the metadata used figure before printing it, eliminating the separate 
> global reserve line, and changing the upthread posted metadata line from 
> 8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the 
> case if the 512 MiB of global reserve indeed comes from the metadata 
> allocation.  This would more clearly show how full metadata actually is 
> without the added complexity of an additional global reserve line, but 
> would lose the fact that global reserve is actually in use, that the 
> broken out global reserve line exposes.
> 
> I'd actually argue in favor of the latter, directly folding global 
> reserve allocation into metadata used, since it'd both be simpler, and 
> more consistent if for instance btrfs fi usage didn't report separate 
> global reserve in the overall stats, but fail to report it in the per-
> device stats and in btrfs dev usage.
> 
> Either way would make much clearer that metadata is actually running out 
> than the current report layout does, since "metadata used" would then 
> either explicitly or implicitly include the global reserve.
> 


-- 
With best regards,
Dmitry
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Austin S Hemmelgarn]

On 2015-11-19 21:11, Duncan wrote:

Austin S Hemmelgarn posted on Thu, 19 Nov 2015 07:28:34 -0500 as
excerpted:


(having all updates installed on Ubuntu doesn't really count in this
case, they're pretty bad sometimes about not properly tracking upstream
development[)]


No kidding.  I'm involved with an upstream that had a security patch and
version-bump a number of years ago.  An Ubuntu bug was filed... and it
sat in the bug queue IIRC untouched until it was obsoleted by newer
releases, where the new version /was/ included.  At least Fedora (which I
remember a poster confirming the update on) and Gentoo (which I run,
personally filed a bug with, and saw the security bump) made the version
bump available as an update.

Apparently, as it wasn't a headline component (one would /hope/ they at
least get security updates out for /them/, and they evidently do for at
least some as they do publish security updates, but at this point I'd
wonder how consistently they do for others), Ubuntu simply didn't care.
Made /me/ glad I wasn't on Ubuntu, that's for sure!

Yeah, that's one of the reasons that I switched to Gentoo.  For those 
who may be interested, the full list is:
1. Way easier to stay up to date (from an administrative perspective, 
not necessarily a computational time perspective (although emerge does 
compute dependencies noticeably faster than dnf, yum, and zypper most of 
the time)).
2. Exponentially more configurable than almost any other full distro 
(this is the big one that really made me choose Gentoo initially).
3. I build my own kernels, and Gentoo has direct integration for this 
(in fact, they only distribute the sources for the kernel, you build it 
locally regardless (which is really not hard even if you don't use 
genkernel to automate it)).
4. Better response to bug reports (as compared to Ubuntu, although this 
is on average, and not always consistent).
5. Proper support for a wide variety of desktop environments in the main 
project (In almost every other distro, any desktop other than the 
default is usually an after thought, or maintained through a separate 
project; and, Xfce (which is what I use) is not very popular as a 
default environment for some reason).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Duncan]

linux-btrfs.tebulin posted on Fri, 20 Nov 2015 10:38:33 +0100 as
excerpted:

> Am 20.11.2015 um 04:14 schrieb Duncan:
>> Meta-comment:
>> 
>> Apparently that attribution should actually be to Hugo Mills.  I've no
>> idea what went wrong, but at least here as received from gmane.org, the
>> from header really does say linux-btrfs.tebulin, so something obviously
>> bugged out somewhere!
> 
> Meta-Answer: My fault
> 
> I accidentally exposed my spamgourmet proxy address to Hugo by taking
> him CC. He replied to all assuming it would go directly to the list, but
> the proxy address masked his original address an put id in it.
> 
> Sorry for messing up!

Makes sense.

While I suppose his reply to all went to the list and to the proxy, the 
proxy bounced it, presumably with the same message-id, and anything that 
tracks by message-id (including my client), which is supposed to be 
unique and thus usable for message tracking, would see only one of the 
two.  Which one would depend on which one got to it first, and whether it 
took that and refused the other one, or let the second one override the 
first.

Either way, definitely confusing, but at least there's a reasonable 
explanation now.  Thanks for helping to clear that up! =:^)

(FWIW, as with most linux kernel lists, replying to all is customary.  I 
generally reply only to the list, however, because it's far easier with 
my specific setup, tho I'll reply to author as well if specifically 
requested.  But Hugo actually does the list-recommended thing by replying 
to all by default.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Dmitry Katsubo]

On 2015-11-20 14:52, Austin S Hemmelgarn wrote:
> On 2015-11-20 08:27, Hugo Mills wrote:
>> On Fri, Nov 20, 2015 at 08:21:31AM -0500, Austin S Hemmelgarn wrote:
>>> On 2015-11-20 06:39, Dmitry Katsubo wrote:
 For those power users who really want to see the tiny details like
 "System" and "GlobalReserve" I suggest to implement "-v" flag:

 # btrfs fi usage -v
>>> Actually, I really like this idea, one of the questions I get asked
>>> when I show people BTRFS is the difference between System and
>>> Metadata, and it's not always easy to explain to somebody who
>>> doesn't have a background in filesystem development.  For some
>>> reason, people seem to have trouble with the concept that the system
>>> tree is an index of the other trees.
>>
>> Actually, it's not that in the system chunks. :)
>>
>> System chunks contain the chunk tree, not the tree of tree roots.
>> They're special (and small) because they're listed explicitly by devid
>> and physical offset at the end of the superblock, and allow the FS to
>> read them first so that it can bootstrap the logical:physical mapping
>> table before it starts reading all the other metadata like the tree of
>> tree roots (which is "normal" metadata).
> I guess my understanding was wrong then.  Thanks for the explanation.

The size of "System" is anyway very small in comparison other types of
allocations. Adding it to "Metadata" or simply suppressing does not make
big difference. If shown, it actually raises "dummy" questions (Is
"System" area big enough? Can it run out of free space? How can I add
more space to it?).

-- 
With best regards,
Dmitry
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Patrik Lundquist]

On 19 November 2015 at 06:58, Roman Mamedov  wrote:
>
> On Wed, 18 Nov 2015 19:53:03 +0100
> linux-btrfs.tebu...@xoxy.net wrote:
>
> >   $ uname -a
> >   Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8
> > 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[...]
>
> So my suggestion would be to try a newer kernel from www.kernel.org: if the
> problem disappears at 4.1 then just keep on using that, or 4.3 if you have to,
> but otherwise that one might be a bit too new to start using right away.

Give http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/ a try.

wget -e robots=off -r -l1 -np -nd -A '*all.deb','*generic*amd64.deb'
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[linux-btrfs . tebulin]

This explanation helped a lot. Got it now!

Thank you!

>You start with a load of unused space -- that's the "total" for
> each device in btrfs fi show. That space is allocated to specific
> usages as the FS needs it. The allocated space is the "used" in btrfs
> fi show for each device.
> 
>Then we switch to looking at btrfs fi df. The "total" values are
> the amount of the *allocated* space given to each type of storage.
> Within those, actual stuff is stored (like your files), which is the
> "used" value for each kind of storage.
> 
>If you've ever played SimCity, the allocation process is like
> zoning -- you say what kind of thing can go on the space, but it's not
> actually used until something gets built on it.
> 
>The problem you hit is when everything has been allocated, and
> there's a need for more metadata space (usually), and there's lots of
> unused data space. The balance operation moves some things around so
> that some of the unused data allocation can be freed up, giving the FS
> the ability to allocate more metadata space.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[linux-btrfs . tebulin]

>It's just freed up lots of space. You'll probably find that your
> "total" value for data in btrfs fi df is close to (but not exactly) 66
> GiB now, if you've just run a full unfiltered balance. (The difference
> being made up of metadata).

I think i need to dive more into the details of btrfs to finally grasp
the details here

>There's a repo of "useful btrfs scripts" that David Sterba looks
> after. 

Great pointer!
I think you refer to https://github.com/kdave/btrfsmaintenance and it
indeed looks very promising.

>> Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an 
>> impending "disk full" situation?  
>In the general case, you need both of them to make sense of it. 

Duh I'll better... ehh... try to find some backported btrfs-profs
supporting `usage`.

>> Will newer kernels do the balance on their own? 
>I think it's on the "projects" list on the wiki [..]  aware of anyone 
> working on it.

Ok - which is another +1 for looking at David 's repo.

Thanks!
- Ben

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[linux-btrfs . tebulin]

On Thu, Nov 19, 2015 at 07:45:13PM +0100, Ben Tebulin wrote:
> 
> >It's just freed up lots of space. You'll probably find that your
> > "total" value for data in btrfs fi df is close to (but not exactly) 66
> > GiB now, if you've just run a full unfiltered balance. (The difference
> > being made up of metadata).
> 
> I think i need to dive more into the details of btrfs to finally grasp
> the details here
> 
> >There's a repo of "useful btrfs scripts" that David Sterba looks
> > after. 
> 
> Great pointer!
> I think you refer to https://github.com/kdave/btrfsmaintenance and it
> indeed looks very promising.
> 
> >> Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an 
> >> impending "disk full" situation?  
> >In the general case, you need both of them to make sense of it. 
> 
> Duh I'll better... ehh... try to find some backported btrfs-profs
> supporting `usage`.

   It's not massively more helpful -- you still need to understand the
relationships. It just uses slightly different language, and puts all
the information in one output instead of two.

   You start with a load of unused space -- that's the "total" for
each device in btrfs fi show. That space is allocated to specific
usages as the FS needs it. The allocated space is the "used" in btrfs
fi show for each device.

   Then we switch to looking at btrfs fi df. The "total" values are
the amount of the *allocated* space given to each type of storage.
Within those, actual stuff is stored (like your files), which is the
"used" value for each kind of storage.

   If you've ever played SimCity, the allocation process is like
zoning -- you say what kind of thing can go on the space, but it's not
actually used until something gets built on it.

   The problem you hit is when everything has been allocated, and
there's a need for more metadata space (usually), and there's lots of
unused data space. The balance operation moves some things around so
that some of the unused data allocation can be freed up, giving the FS
the ability to allocate more metadata space.

   Hugo.

> >> Will newer kernels do the balance on their own? 
> >I think it's on the "projects" list on the wiki [..]  aware of anyone 
> > working on it.
> 
> Ok - which is another +1 for looking at David 's repo.
> 
> Thanks!
> - Ben

-- 
Hugo Mills | 2 + 2 = 5, for sufficiently large values of 2.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Austin S Hemmelgarn]

On 2015-11-19 13:28, Hugo Mills wrote:

On Thu, Nov 19, 2015 at 06:35:24PM +0100, linux-btrfs.tebu...@xoxy.net wrote:

Will newer kernels do the balance on their own?


I think it's on the "projects" list on the wiki, so it may get done
eventually. As I said above, I'm not aware of anyone working on it.
TBH, while it would be a nice feature, it's also something that has the 
potential to cause issues for people if it all happens at once.  IN many 
of the cases I've seen, the usual issue is lots of mostly empty data 
chunks (usually less than 20% in the two times I've run into this 
myself).  Based on this, having something in the kernel that could scan 
chunk usage every now and then, and repack mostly empty chunks if there 
is space already allocated in existing chunks, would probably 
significantly reduce the chances of this happening for most people, and 
cause much less noticeable performance degradation than triggering a 
full balance on a chunk allocation failure.  Since I started using a 
cron job to do a daily balance of chunks less than 20% full, and a 
weekly one for chunks less than 50% full, I've not run into these issues 
at all, and actually see on average better performance on my tradition 
hard disks.





smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Austin S Hemmelgarn]

On 2015-11-18 13:53, linux-btrfs.tebu...@xoxy.net wrote:


P.S.: Just as user feedback: For /srv I'm using on the very same system
ZFS since the very first day. With snapshots & all the fancy stuff like
ZRAID-1, lz4, ... My number of Issues there: 0
Since other people have adequately answered the main questions, I'll 
just comment on this.


In general, ZFS is not a good comparison to BTRFS for anything but 
features.  Just the open source version of ZFS has been around for more 
than a decade, and it existed as proprietary code for quite a while 
before that.  As such, it's had a lot longer to stabilize, has had 
exponentially more testing, and in general is a lot more reliable.  As 
such, aside from feature comparisons, ZFS is going to win in almost all 
tests of performance, reliability, and usability for the foreseeable 
future.  This doesn't mean you shouldn't use BTRFS (ZFS got to where it 
is now exactly because _a lot_ of people use it), it just means you need 
to do a lot of homework, and ideally keep you system actually up to date 
(having all updates installed on Ubuntu doesn't really count in this 
case, they're pretty bad sometimes about not properly tracking upstream 
development (although they are significantly better than regular Debian)).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Duncan]

Austin S Hemmelgarn posted on Thu, 19 Nov 2015 07:28:34 -0500 as
excerpted:

> (having all updates installed on Ubuntu doesn't really count in this
> case, they're pretty bad sometimes about not properly tracking upstream
> development[)]

No kidding.  I'm involved with an upstream that had a security patch and 
version-bump a number of years ago.  An Ubuntu bug was filed... and it 
sat in the bug queue IIRC untouched until it was obsoleted by newer 
releases, where the new version /was/ included.  At least Fedora (which I 
remember a poster confirming the update on) and Gentoo (which I run, 
personally filed a bug with, and saw the security bump) made the version 
bump available as an update.

Apparently, as it wasn't a headline component (one would /hope/ they at 
least get security updates out for /them/, and they evidently do for at 
least some as they do publish security updates, but at this point I'd 
wonder how consistently they do for others), Ubuntu simply didn't care.  
Made /me/ glad I wasn't on Ubuntu, that's for sure!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Duncan]

linux-btrfs.tebulin posted on Thu, 19 Nov 2015 18:56:45 + as
excerpted:

Meta-comment:

Apparently that attribution should actually be to Hugo Mills.  I've no 
idea what went wrong, but at least here as received from gmane.org, the 
from header really does say linux-btrfs.tebulin, so something obviously 
bugged out somewhere!


Meanwhile discussing btrfs data/metadata allocation, vs. usage of that 
allocation, Hugo also known here as tebulin, explained...

> If you've ever played SimCity, the allocation process is like zoning --
> you say what kind of thing can go on the space, but it's not actually
> used until something gets built on it.

Very nice analogy.  Thanks. =:^)

Tho I'd actually put it in terms of the real thing that sim city 
simulates, thus eliminating the proprietary comparison.  A city zones an 
area one way or another, restricting what can be built there -- you can't 
put heavy industry in a residential zone.  But the lot is still empty 
until something's actually built there.

And if the city has all its area zoned residential, and some company 
wants to build a plant there (generally considered a good thing as it'll 
provide employment), there's a process that allows rezoning.

In btrfs, there's four types of allocations aka "zones":

1) Unallocated (unzoned)

Can be used for anything but must be allocated/zoned first

2) System

Critical but limited in size and generally only allocated at mkfs or when 
adding a new device.

3) Data

The actual file storage, generally the largest allocation.

4) Metadata

Information /about/ the files, where they are located (the location and 
size of individual extents), ownership and permissions, date stamps, 
checksums, and for very small files (a few KiB), sometimes the file 
content itself.

4a) Global reserve

A small part of metadata reserved for emergency use only.  Btrfs is 
pretty strict about its use, and will generally error out with ENOSPC if 
metadata space other than the global reserve is used up, before actually 
using this global reserve.  As a result, any of the global reserve used 
at all indicates a filesystem in very severe straits, crisis mode.


As it happens, btrfs in practice tends to be a bit liberal about 
allocating/zoning data chunks, since it's easier to find bigger blocks of 
space in unallocated space than it is in busy partly used data space.  
(Think of a big shopping center.  It's easier to build it in outlying 
areas that haven't been built up yet, where many whole blocks worth of 
space can be allocated/zoned at once, than it is in the city center, 
where even finding a single whole block vacant, is near impossible.)

Over time, therefore, more and more space tends to be allocated to data, 
while existing data space, like those blocks near city center, may have 
most of its files/buildings deleted, but still have a couple still 
occupied.

Btrfs balancing, then, is comparable to the city functions of 
condemnation and rezoning to vacant/unallocated, forcing remaining 
occupants, most commonly data zoned, to move out of the way so the area 
can be reclaimed for other usage.  Then it can be rezoned to data again, 
or to metadata, whatever needs it.


(FWIW, I played the original sim city, but IIRC it wasn't sophisticated 
enough to have zoning yet.  Of course I've not played anything recent as 
to my knowledge it's not freedomware, and since I no longer agree to 
waive rights via EULA, etc, I can no longer legally install and run such 
things.  But the real life sim city simulates, is the antitype behind 
both that simulation, and the analogy I drew here, based on the one you 
drew to the simulation.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Hugo Mills]

On Wed, Nov 18, 2015 at 07:53:03PM +0100, linux-btrfs.tebu...@xoxy.net wrote:
> Hello there!
> 
> I'm stumbling over this regularly. I explicitly upgraded the kernel to
> avoid this, but it still occurs me every few months:
> 
>   $ touch foo
>   touch: »foo“ kann nicht berührt werden: Auf dem Gerät ist kein
> Speicherplatz mehr verfügbar
>   $ sudo btrfs fi df /
>   Data, single: total=101.14GiB, used=78.99GiB
>   System, DUP: total=32.00MiB, used=16.00KiB
>   Metadata, DUP: total=9.00GiB, used=8.48GiB
>   unknown, single: total=512.00MiB, used=832.00KiB
> 
> Whut? Why? WTF?

   Running out of metadata space, probably, although it's impossible
to tell from only this information -- see below.

>   $ uname -a
>   Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8
> 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> dmesg is empty. Resolution is to delete a whole bunch of btrfs
> subvolumes which are produced by apt-btrfs-snapshot. I run a
>sudo btrfs balance start -dusage=5 /
> afterwards as part of my regular "i have no clue what the heck is wrong
> with this; maybe it will help & mend my problem" process.
> 
> This is annoying. Why is even btrfs own "disk free" utility lying to me?

   I't snot lying to you -- you're just not seeing the whole
story. You can't tell everything about disk usage from btrfs fi df.
You need to use btrfs fi show as well to get the full information (or
btrfs fi usage, which combines the output of both). See the FAQ[1] for
how to interpret the output.

> How to fix this? How to avoid this? How to get notified about this?

   Without seeing your btrfs fi show output, I can't say for sure, but
the usual problem in this instance is that you've got all your
available space allocated to either data or metadata, and the FS needs
more metadata space and can't allocate it. The solution is to free up
some of the data allocation, using a filtered balance. See the FAQ[2]
for how to spot the problem, and what to do about it.

[1] 
https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools
[2] 
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

   Hugo.

> - Ben
> 
> P.S.: Just as user feedback: For /srv I'm using on the very same system
> ZFS since the very first day. With snapshots & all the fancy stuff like
> ZRAID-1, lz4, ... My number of Issues there: 0
> 

-- 
Hugo Mills | Close enough for government work.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Qu Wenruo]

Hugo Mills wrote on 2015/11/18 19:08 +:

On Wed, Nov 18, 2015 at 07:53:03PM +0100, linux-btrfs.tebu...@xoxy.net wrote:

Hello there!

I'm stumbling over this regularly. I explicitly upgraded the kernel to
avoid this, but it still occurs me every few months:

   $ touch foo
   touch: »foo“ kann nicht berührt werden: Auf dem Gerät ist kein
Speicherplatz mehr verfügbar
   $ sudo btrfs fi df /
   Data, single: total=101.14GiB, used=78.99GiB
   System, DUP: total=32.00MiB, used=16.00KiB
   Metadata, DUP: total=9.00GiB, used=8.48GiB
   unknown, single: total=512.00MiB, used=832.00KiB

Whut? Why? WTF?


Running out of metadata space, probably, although it's impossible
to tell from only this information -- see below.


It's almost 100% sure that you already run out of metadata space.

Although the metadata output is showing that you still have about 512M 
available, but the 512M is Global Reserved space, or the unknown one.


Global reserved space should not be used, until metadata is really 
running out.
And in your case, you are already using global reserved space, see the 
832KB?


You may question that why not continue using global reserved space, but 
that's the last method, so Btrfs will just give you ENOSPC error.



The output is really a little confusing. I'd like the change the output 
by adding global reserved into metadata used space and make it a sub 
item for metadata.

This should make the output more understandable.

Thanks,
Qu




   $ uname -a
   Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8
10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

dmesg is empty. Resolution is to delete a whole bunch of btrfs
subvolumes which are produced by apt-btrfs-snapshot. I run a
sudo btrfs balance start -dusage=5 /
afterwards as part of my regular "i have no clue what the heck is wrong
with this; maybe it will help & mend my problem" process.

This is annoying. Why is even btrfs own "disk free" utility lying to me?


I't snot lying to you -- you're just not seeing the whole
story. You can't tell everything about disk usage from btrfs fi df.
You need to use btrfs fi show as well to get the full information (or
btrfs fi usage, which combines the output of both). See the FAQ[1] for
how to interpret the output.


How to fix this? How to avoid this? How to get notified about this?


Without seeing your btrfs fi show output, I can't say for sure, but
the usual problem in this instance is that you've got all your
available space allocated to either data or metadata, and the FS needs
more metadata space and can't allocate it. The solution is to free up
some of the data allocation, using a filtered balance. See the FAQ[2]
for how to spot the problem, and what to do about it.

[1] 
https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools
[2] 
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

Hugo.


- Ben

P.S.: Just as user feedback: For /srv I'm using on the very same system
ZFS since the very first day. With snapshots & all the fancy stuff like
ZRAID-1, lz4, ... My number of Issues there: 0




--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Duncan]

Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:

> Although the metadata output is showing that you still have about 512M
> available, but the 512M is Global Reserved space, or the unknown one.

Unknown here, as the userspace (btrfs-progs) is evidently too old to show 
it as global reserve, as it does in newer versions...

> The output is really a little confusing. I'd like the change the output
> by adding global reserved into metadata used space and make it a sub
> item for metadata.

Thanks for the clarification.  It's most helpful, here. =:^)

I've at times wondered if global reserve folded into one of the other 
settings.  Apparently it comes from the metadata allocation, but while 
metadata is normally dup (single-device btrfs) or raid1 (multi-device), 
global reserve is single.

It would have been nice if that sort of substructure was described a bit 
better when global reserve first made its appearance, at least in the 
patch descriptions and release announcement, if not then yet in btrfs fi 
df output, first implementations being what they are.  But regardless, 
now at least it should be clear for list regulars who read this thread 
anyway, since the above reasonably clarifies things.

As for btrfs fi df, making global reserve a metadata subentry there would 
be one way to deal with it, preserving the exposure of the additional 
data provided by that line (here, the fact that global reserve is 
actually being used, underlining the fact that the filesystem is severely 
short on space).

Another way of handling it would be to simply add the global reserve into 
the metadata used figure before printing it, eliminating the separate 
global reserve line, and changing the upthread posted metadata line from 
8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the 
case if the 512 MiB of global reserve indeed comes from the metadata 
allocation.  This would more clearly show how full metadata actually is 
without the added complexity of an additional global reserve line, but 
would lose the fact that global reserve is actually in use, that the 
broken out global reserve line exposes.

I'd actually argue in favor of the latter, directly folding global 
reserve allocation into metadata used, since it'd both be simpler, and 
more consistent if for instance btrfs fi usage didn't report separate 
global reserve in the overall stats, but fail to report it in the per-
device stats and in btrfs dev usage.

Either way would make much clearer that metadata is actually running out 
than the current report layout does, since "metadata used" would then 
either explicitly or implicitly include the global reserve.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

[Roman Mamedov]

On Wed, 18 Nov 2015 19:53:03 +0100
linux-btrfs.tebu...@xoxy.net wrote:

>   $ uname -a
>   Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8
> 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

For whatever unknown reason, Ubuntu repeatedly chooses to stabilize on exactly
the wrong kernel versions. Their recent releases were based on kernels 3.11,
3.13 and 3.19. None of these were chosen by the upstream as the "longterm"
branches, and as you can see none are listed anymore on https://www.kernel.org/

To emphasize -- the mainline Linux kernel developers (and Btrfs developers)
basically do not care anymore about 3.19 at all. What you currently use, is a
piecemeal of unknown quality, cobbled together by Ubuntu developers from
patches lifted either from 3.18 or 4.1 series -- by taking fixes they deem
important and frankensteining them together until they somehow seem to apply.

And do they back/forward-port Btrfs-related bugfixes?... all of them?... are
they doing that well enough? Just nobody knows.

So my suggestion would be to try a newer kernel from www.kernel.org: if the
problem disappears at 4.1 then just keep on using that, or 4.3 if you have to,
but otherwise that one might be a bit too new to start using right away.

-- 
With respect,
Roman


signature.asc
Description: PGP signature