Re: Missing half of available space (resend)

[David Hampton]

On Wed, 2015-12-09 at 16:48 +, Duncan wrote:
> David Hampton posted on Wed, 09 Dec 2015 01:30:09 -0500 as excerpted:
> 
> > Seems I need to upgrade my tools.  That command was added in 3.18 and I
> > only have the 3.12 tools.
> 
> Definitely so, especially because you're running raid6, which wasn't 
> stable until 4.1 for both kernel and userspace.  3.12?  I guess it did 
> have the very basic raid56 support, but it's definitely nothing I'd 
> trust, at that old not for btrfs in general, but FOR SURE not raid56.

I've upgraded to the 4.2.0 kernel and the 4.0 btrfs-tools package.
These are the latest that Ubuntu has packaged for 15.10, and I've pulled
them into my 14.10 based release.  Is this recent enough, or do I need
to try the 4.3 kernel/tools build from the active development tree (that
will eventually become 16.04)?

David



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[David Hampton]

This is a Mythbuntu system, and the latest they support is 14.04. 

Thanks for all the responses.

David

On Thu, 2015-12-10 at 06:56 +1100, Gareth Pye wrote:
> I wouldn't blame Ubuntu too much, 14.10 went out of support months ago
> (which counts as a long time when it's only for people happy to
> upgrade every 6 months).
> 
> The kernel ppa's builds tend to run fine on the latest LTS & regular
> releases, although they can cause issues (I've had some fun with
> nvidia drivers at times). That ppa will get you to 4.3 or 4.4rc4.
> 
> On Thu, Dec 10, 2015 at 6:39 AM, Chris Murphy  wrote:
> > On Wed, Dec 9, 2015 at 10:28 AM, David Hampton
> >  wrote:
> >> On Wed, 2015-12-09 at 16:48 +, Duncan wrote:
> >>> David Hampton posted on Wed, 09 Dec 2015 01:30:09 -0500 as excerpted:
> >>>
> >>> > Seems I need to upgrade my tools.  That command was added in 3.18 and I
> >>> > only have the 3.12 tools.
> >>>
> >>> Definitely so, especially because you're running raid6, which wasn't
> >>> stable until 4.1 for both kernel and userspace.  3.12?  I guess it did
> >>> have the very basic raid56 support, but it's definitely nothing I'd
> >>> trust, at that old not for btrfs in general, but FOR SURE not raid56.
> >>
> >> I've upgraded to the 4.2.0 kernel and the 4.0 btrfs-tools package.
> >
> > I think btrfs-progs 4.0 has a mkfs bug in it (or was that 4.0.1?)
> > Anyway, even that is still old in Btrfs terms. I think Ubuntu needs to
> > do better than this, or just acknowledge Btrfs is not supported, don't
> > include btrfs-progs at all by default, and stop making it an install
> > time option.
> >
> >
> >> These are the latest that Ubuntu has packaged for 15.10, and I've pulled
> >> them into my 14.10 based release.  Is this recent enough, or do I need
> >> to try the 4.3 kernel/tools build from the active development tree (that
> >> will eventually become 16.04)?
> >
> > It's probably fine day to day, but if you ever were to need btrfs
> > check or repair, you'd want the current version no matter what. There
> > are just too many bug fixes and enhancements happening to not make
> > that effort. You kinda have to understand that you're effectively
> > testing Btrfs by using raid56. It is stabilizing, but it can hardly be
> > called stable or even feature complete seeing as there are all sorts
> > of missing failure notifications.
> >
> > More than anything else you need to be willing to lose everything on
> > this volume, without further notice, i.e. you need a backup strategy
> > that you're prepared to use without undue stress. If you can't do
> > that, you need to look at another arrangement. Both LVM and mdadm
> > raid6 + XFS are more stable.
> >
> >
> > --
> > Chris Murphy
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[David Hampton]

Ubuntu 14.04 actually ships with the 3.13 kernel.  I had already
upgraded it to 3.19 from the Ubuntu 15.04 release.

I'm pretty sure I created the btrfs partition, not the MythBuntu
installer.  I don't remember if that was even an option.

David


On Wed, 2015-12-09 at 14:28 -0700, Chris Murphy wrote:
> On Wed, Dec 9, 2015 at 12:56 PM, Gareth Pye  wrote:
> > I wouldn't blame Ubuntu too much, 14.10 went out of support months ago
> 
> OP reported 3.19.0-32-generic #37~14.04.1-Ubuntu. And 14.04 is LTS
> supported until 2019. I think it should have something newer for both
> kernel and progs, if it's going to offer btrfs as an install time
> option. It's really easy to just have the LTS installer not offer
> Btrfs, and not install btrfs-progs.
> 
> 
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[Chris Murphy]

On Wed, Dec 9, 2015 at 12:56 PM, Gareth Pye  wrote:
> I wouldn't blame Ubuntu too much, 14.10 went out of support months ago

OP reported 3.19.0-32-generic #37~14.04.1-Ubuntu. And 14.04 is LTS
supported until 2019. I think it should have something newer for both
kernel and progs, if it's going to offer btrfs as an install time
option. It's really easy to just have the LTS installer not offer
Btrfs, and not install btrfs-progs.



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[Gareth Pye]

Sorry his message 4 hours ago mentioned 14.10.

On Thu, Dec 10, 2015 at 8:41 AM, David Hampton
 wrote:
> Ubuntu 14.04 actually ships with the 3.13 kernel.  I had already
> upgraded it to 3.19 from the Ubuntu 15.04 release.
>
> I'm pretty sure I created the btrfs partition, not the MythBuntu
> installer.  I don't remember if that was even an option.
>
> David
>
>
> On Wed, 2015-12-09 at 14:28 -0700, Chris Murphy wrote:
>> On Wed, Dec 9, 2015 at 12:56 PM, Gareth Pye  wrote:
>> > I wouldn't blame Ubuntu too much, 14.10 went out of support months ago
>>
>> OP reported 3.19.0-32-generic #37~14.04.1-Ubuntu. And 14.04 is LTS
>> supported until 2019. I think it should have something newer for both
>> kernel and progs, if it's going to offer btrfs as an install time
>> option. It's really easy to just have the LTS installer not offer
>> Btrfs, and not install btrfs-progs.
>>
>>
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[Chris Murphy]

On Wed, Dec 9, 2015 at 10:28 AM, David Hampton
 wrote:
> On Wed, 2015-12-09 at 16:48 +, Duncan wrote:
>> David Hampton posted on Wed, 09 Dec 2015 01:30:09 -0500 as excerpted:
>>
>> > Seems I need to upgrade my tools.  That command was added in 3.18 and I
>> > only have the 3.12 tools.
>>
>> Definitely so, especially because you're running raid6, which wasn't
>> stable until 4.1 for both kernel and userspace.  3.12?  I guess it did
>> have the very basic raid56 support, but it's definitely nothing I'd
>> trust, at that old not for btrfs in general, but FOR SURE not raid56.
>
> I've upgraded to the 4.2.0 kernel and the 4.0 btrfs-tools package.

I think btrfs-progs 4.0 has a mkfs bug in it (or was that 4.0.1?)
Anyway, even that is still old in Btrfs terms. I think Ubuntu needs to
do better than this, or just acknowledge Btrfs is not supported, don't
include btrfs-progs at all by default, and stop making it an install
time option.


> These are the latest that Ubuntu has packaged for 15.10, and I've pulled
> them into my 14.10 based release.  Is this recent enough, or do I need
> to try the 4.3 kernel/tools build from the active development tree (that
> will eventually become 16.04)?

It's probably fine day to day, but if you ever were to need btrfs
check or repair, you'd want the current version no matter what. There
are just too many bug fixes and enhancements happening to not make
that effort. You kinda have to understand that you're effectively
testing Btrfs by using raid56. It is stabilizing, but it can hardly be
called stable or even feature complete seeing as there are all sorts
of missing failure notifications.

More than anything else you need to be willing to lose everything on
this volume, without further notice, i.e. you need a backup strategy
that you're prepared to use without undue stress. If you can't do
that, you need to look at another arrangement. Both LVM and mdadm
raid6 + XFS are more stable.


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[Gareth Pye]

I wouldn't blame Ubuntu too much, 14.10 went out of support months ago
(which counts as a long time when it's only for people happy to
upgrade every 6 months).

The kernel ppa's builds tend to run fine on the latest LTS & regular
releases, although they can cause issues (I've had some fun with
nvidia drivers at times). That ppa will get you to 4.3 or 4.4rc4.

On Thu, Dec 10, 2015 at 6:39 AM, Chris Murphy  wrote:
> On Wed, Dec 9, 2015 at 10:28 AM, David Hampton
>  wrote:
>> On Wed, 2015-12-09 at 16:48 +, Duncan wrote:
>>> David Hampton posted on Wed, 09 Dec 2015 01:30:09 -0500 as excerpted:
>>>
>>> > Seems I need to upgrade my tools.  That command was added in 3.18 and I
>>> > only have the 3.12 tools.
>>>
>>> Definitely so, especially because you're running raid6, which wasn't
>>> stable until 4.1 for both kernel and userspace.  3.12?  I guess it did
>>> have the very basic raid56 support, but it's definitely nothing I'd
>>> trust, at that old not for btrfs in general, but FOR SURE not raid56.
>>
>> I've upgraded to the 4.2.0 kernel and the 4.0 btrfs-tools package.
>
> I think btrfs-progs 4.0 has a mkfs bug in it (or was that 4.0.1?)
> Anyway, even that is still old in Btrfs terms. I think Ubuntu needs to
> do better than this, or just acknowledge Btrfs is not supported, don't
> include btrfs-progs at all by default, and stop making it an install
> time option.
>
>
>> These are the latest that Ubuntu has packaged for 15.10, and I've pulled
>> them into my 14.10 based release.  Is this recent enough, or do I need
>> to try the 4.3 kernel/tools build from the active development tree (that
>> will eventually become 16.04)?
>
> It's probably fine day to day, but if you ever were to need btrfs
> check or repair, you'd want the current version no matter what. There
> are just too many bug fixes and enhancements happening to not make
> that effort. You kinda have to understand that you're effectively
> testing Btrfs by using raid56. It is stabilizing, but it can hardly be
> called stable or even feature complete seeing as there are all sorts
> of missing failure notifications.
>
> More than anything else you need to be willing to lose everything on
> this volume, without further notice, i.e. you need a backup strategy
> that you're prepared to use without undue stress. If you can't do
> that, you need to look at another arrangement. Both LVM and mdadm
> raid6 + XFS are more stable.
>
>
> --
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[Duncan]

David Hampton posted on Wed, 09 Dec 2015 01:30:09 -0500 as excerpted:

> Seems I need to upgrade my tools.  That command was added in 3.18 and I
> only have the 3.12 tools.

Definitely so, especially because you're running raid6, which wasn't 
stable until 4.1 for both kernel and userspace.  3.12?  I guess it did 
have the very basic raid56 support, but it's definitely nothing I'd 
trust, at that old not for btrfs in general, but FOR SURE not raid56.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Missing half of available space (resend)

[David Hampton]

Hi all.  I'm trying to figuring out why my btrfs file system doesn't
show all the available space.  I currently have four 4TB drives set up
as a raid6 array, so I would expect to see a total available data size
slightly under 8TB (two drives for data + two drives for parity).  The
'btrfs fi df' command consistently shows a total size of around 3TB, and
says that space is almost completely full.  Here's my current system
information...

===
root@selene:~# uname -a
Linux selene.dhampton.net 3.19.0-32-generic #37~14.04.1-Ubuntu SMP Thu
Oct 22 09:41:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@selene:~# btrfs --version
Btrfs v3.12
root@selene:~# btrfs fi show /video
Label: none  uuid: 74a4c4fa-9e83-465a-850d-cc089ecd00f6
Total devices 4 FS bytes used 3.12TiB
devid1 size 3.64TiB used 1.58TiB path /dev/vdb
devid2 size 3.64TiB used 1.58TiB path /dev/vda
devid3 size 3.64TiB used 1.58TiB path /dev/vdc
devid4 size 3.64TiB used 1.58TiB path /dev/vdd

Btrfs v3.12
root@selene:~# btrfs fi df /video
Data, RAID6: total=3.15TiB, used=3.11TiB
System, RAID6: total=64.00MiB, used=352.00KiB
Metadata, RAID6: total=5.00GiB, used=3.73GiB
unknown, single: total=512.00MiB, used=1.07MiB
root@selene:~# df -h /video
Filesystem  Size  Used Avail Use% Mounted on
/dev/vda 15T  3.2T  8.3T  28% /video
===

I have tried issuing the command "btrfs filesystem resize
:max /video" on each devid in the array, and also tried balancing
the array.  None of these commands changed the indication that the file
system is almost full.  I'm wondering if the problem is because this
file system began as a two drive raid1 array, and I later added the
other two drives and used the 'btrfs balance' command to convert to
raid6.  Any suggestions on what I can try to get the 'btrfs fi df'
command to show me more available space?  Did I forget a command when I
converted the raid1 array to raid6?

Alternatively, can I trust the numbers in the standard df command?  The
'used' number seems right but the 'avail' number seems high.

If i can provide any more information to help figure out what's
happening, please ask.

Thanks.

David


[0.00] Initializing cgroup subsys cpuset
[0.00] Initializing cgroup subsys cpu
[0.00] Initializing cgroup subsys cpuacct
[0.00] Linux version 3.19.0-32-generic (buildd@lgw01-43) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #37~14.04.1-Ubuntu SMP Thu Oct 22 09:41:40 UTC 2015 (Ubuntu 3.19.0-32.37~14.04.1-generic 3.19.8-ckt7)
[0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-32-generic root=UUID=b9fb1104-f681-4664-b0c3-b17db28d9d68 ro quiet splash vt.handoff=7
[0.00] KERNEL supported cpus:
[0.00]   Intel GenuineIntel
[0.00]   AMD AuthenticAMD
[0.00]   Centaur CentaurHauls
[0.00] e820: BIOS-provided physical RAM map:
[0.00] BIOS-e820: [mem 0x-0x0009fbff] usable
[0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved
[0.00] BIOS-e820: [mem 0x000f-0x000f] reserved
[0.00] BIOS-e820: [mem 0x0010-0x3fffdfff] usable
[0.00] BIOS-e820: [mem 0x3fffe000-0x3fff] reserved
[0.00] BIOS-e820: [mem 0xfeffc000-0xfeff] reserved
[0.00] BIOS-e820: [mem 0xfffc-0x] reserved
[0.00] NX (Execute Disable) protection: active
[0.00] SMBIOS 2.4 present.
[0.00] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011
[0.00] Hypervisor detected: KVM
[0.00] e820: update [mem 0x-0x0fff] usable ==> reserved
[0.00] e820: remove [mem 0x000a-0x000f] usable
[0.00] AGP: No AGP bridge found
[0.00] e820: last_pfn = 0x3fffe max_arch_pfn = 0x4
[0.00] MTRR default type: write-back
[0.00] MTRR fixed ranges enabled:
[0.00]   0-9 write-back
[0.00]   A-B uncachable
[0.00]   C-F write-protect
[0.00] MTRR variable ranges enabled:
[0.00]   0 base 8000 mask 3FFF8000 uncachable
[0.00]   1 disabled
[0.00]   2 disabled
[0.00]   3 disabled
[0.00]   4 disabled
[0.00]   5 disabled
[0.00]   6 disabled
[0.00]   7 disabled
[0.00] PAT configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- UC  
[0.00] found SMP MP-table at [mem 0x000f1ff0-0x000f1fff] mapped at [880f1ff0]
[0.00] Scanning 1 areas for low memory corruption
[0.00] Base memory trampoline at [88099000] 99000 size 24576
[0.00] init_memory_mapping: [mem 0x-0x000f]
[0.00]  [mem 0x-0x000f] page 4k
[0.00] BRK [0x01fd4000, 0x01fd4fff] PGTABLE
[0.00] BRK [0x01fd5000, 0x01fd5fff] PGTABLE
[0.00] BRK [0x01fd6000, 0x01fd6fff] PGTABLE
[0.00] init_memory_mapping: [mem 

Re: Missing half of available space (resend)

[Chris Murphy]

On Tue, Dec 8, 2015 at 10:02 PM, David Hampton
 wrote:
> The
> 'btrfs fi df' command consistently shows a total size of around 3TB, and
> says that space is almost completely full.

and


> root@selene:~# btrfs fi df /video
> Data, RAID6: total=3.15TiB, used=3.11TiB

The "total=3.15TiB" means "there's a total of 3.15TiB allocated for
data chunks using raid6 profile" and of that 3.11TiB is used.

btrfs fi df doesn't ever show how much is free or available. You can
get an estimate of that by using 'btrfs fi usage' instead.



> root@selene:~# df -h /video
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/vda 15T  3.2T  8.3T  28% /video

That's about right although it seems it's slightly overestimating the
available free space.


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Missing half of available space (resend)

[David Hampton]

On Tue, 2015-12-08 at 22:27 -0700, Chris Murphy wrote:
> On Tue, Dec 8, 2015 at 10:02 PM, David Hampton
>  wrote:
> > The
> > 'btrfs fi df' command consistently shows a total size of around
> > 3TB, and says that space is almost completely full.
> 
> and
> 
> 
> > root@selene:~# btrfs fi df /video
> > Data, RAID6: total=3.15TiB, used=3.11TiB
> 
> The "total=3.15TiB" means "there's a total of 3.15TiB allocated for
> data chunks using raid6 profile" and of that 3.11TiB is used.
> 
> btrfs fi df doesn't ever show how much is free or available.

I think I get it.  The numbers in the 'df' command don't show the total
number of chunks that exist, only the subset of those chunks that have
been allocated to something.

> You can get an estimate of that by using 'btrfs fi usage' instead.

Seems I need to upgrade my tools.  That command was added in 3.18 and I
only have the 3.12 tools.

> > root@selene:~# df -h /video
> > Filesystem  Size  Used Avail Use% Mounted on
> > /dev/vda 15T  3.2T  8.3T  28% /video
> 
> That's about right although it seems it's slightly overestimating the
> available free space.

Thanks.  Make me feel a lot better.

David



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html