Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Austin S. Hemmelgarn]

On 2016-02-11 09:14, Goffredo Baroncelli wrote:

On 2016-02-10 20:59, Austin S. Hemmelgarn wrote:
[...]

Again, a torn write to the metadata referencing the block (stripe in
this case I believe) will result in loosing anything written by the
update to the stripe.


I think that the order matters: first the data block are written (in a new 
location, so the old data are untouched), then the metadata, from the leafs up 
to the upper node (again in a new location), then the superblock which 
references to the upper node of the tree(s).

If you interrupt the writes in any time, the filesystem can survive because the 
old superblock-metadata-tree and data-block are still valid until the last 
pieces (the new superblock) is written.

And if this last step fails, the checksum shows that the super-block is invalid 
and the old one is taken in consideration.
You're not understanding what I'm saying.  If a write fails anywhere 
during the process of updating the metadata, up to and including the 
super-block, then you loose the data writes that triggered the metadata 
update.  This doesn't result in a broken filesystem, but it does result 
in data loss, even if it's not what most people think of as data loss.


To make a really simplified example, assume we have a single block of 
data (D) referenced by a single metadata block (M) and a single 
super-block referencing the metadata block (S).  On a COW filesystem, 
when you write to D, it allocates and writes a new block (D2) to store 
the data, then allocates and writes a new metadata block (M2) to point 
to D2, and then updates the superblock in-place to point to M2.  If the 
write to M2 fails, you loose all new data in D2 that wasn't already in 
D.  There is no way that a COW filesystem can avoid this type of data 
loss without being able to force the underlying storage to atomically 
write out all of D2, M2, and S at the same time, it's an inherent issue 
in COW semantics in general, not just filesystems.




There is no way that _any_ system can avoid
this issue without having the ability to truly atomically write out
the entire metadata tree after the block (stripe) update.


It is not needed to atomically write the (meta)data in a COW filesystem, 
because the new data don't owerwrite the old one. The only thing that is needed 
is that before the last piece is written all the previous (mata)data are 
already written.\
Even when enforcing ordering, the issue I've outlined above is still 
present.  If a write fails at any point in the metadata updates 
cascading up the tree, then any new data below that point in the tree is 
lost.


For not COW filesystem a journal is required to avoid this kind of problem.
To a certain extent yes, but journals have issues that COW doesn't.  A 
torn write in the journal on most traditional journaling filesystems 
will often result in a broken filesystem.



Doing so
would require a degree of tight hardware level integration that's
functionally impossible for any general purpose system (in essence,
the filesystem would have to be implemented in the hardware, not
software).


To solve the raid-write-hole problem, a checksum system (of data and metadata) 
is sufficient. However to protect with checksum the data, it seems that a COW 
filesystem is required.
Either COW, or log structuring, or the ability to atomically write out 
groups of blocks.  Log structuring (like NILFS2, or LogFS, or even LFS 
from *BSD) has performance implications on traditional rotational media, 
and only recently are storage devices appearing that can actually handle 
atomic writes of groups of multiple blocks at the same time, so COW has 
been the predominant model because it works on everything, and doesn't 
have the performance issues of log-structured filesystems (if 
implemented correctly).


The only critical thing, is that the hardware has to not lie about the fact 
that the data reached the platter. Most of the problem reported in the ML are 
related to external disk used in USB enclousure, which most of the time lie 
about this aspect.
That really depends on what you mean by 'lie about the data being on the 
platter'.  All modern hard disks have a write cache, and a decent 
percentage don't properly support flushing the write cache except by 
waiting for it to drain, many of them arbitrarily re-order writes within 
the cache, and none that I've seen have a non-volatile write cache, and 
therefore all such disks arguably lie about when the write is actually 
complete.  SSD's add yet another layer of complexity to this, because 
the good ones have either a non-volatile write cache, or have built-in 
batteries or super-capacitors to make sure they can flush the write 
cache when power is lost, so some SSD's can behave just like HDD's do 
and claim the write is complete when it hits the cache without 
technically lying, but most SSD's don't document whether they do this or 
not.

--
To unsubscribe from this list: send the line "unsubscribe 

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Goffredo Baroncelli]

On 2016-02-10 20:59, Austin S. Hemmelgarn wrote:
[...]
> Again, a torn write to the metadata referencing the block (stripe in
> this case I believe) will result in loosing anything written by the
> update to the stripe. 

I think that the order matters: first the data block are written (in a new 
location, so the old data are untouched), then the metadata, from the leafs up 
to the upper node (again in a new location), then the superblock which 
references to the upper node of the tree(s).

If you interrupt the writes in any time, the filesystem can survive because the 
old superblock-metadata-tree and data-block are still valid until the last 
pieces (the new superblock) is written.

And if this last step fails, the checksum shows that the super-block is invalid 
and the old one is taken in consideration.


> There is no way that _any_ system can avoid
> this issue without having the ability to truly atomically write out
> the entire metadata tree after the block (stripe) update.  

It is not needed to atomically write the (meta)data in a COW filesystem, 
because the new data don't owerwrite the old one. The only thing that is needed 
is that before the last piece is written all the previous (mata)data are 
already written.

For not COW filesystem a journal is required to avoid this kind of problem.

> Doing so
> would require a degree of tight hardware level integration that's
> functionally impossible for any general purpose system (in essence,
> the filesystem would have to be implemented in the hardware, not
> software).

To solve the raid-write-hole problem, a checksum system (of data and metadata) 
is sufficient. However to protect with checksum the data, it seems that a COW 
filesystem is required.

The only critical thing, is that the hardware has to not lie about the fact 
that the data reached the platter. Most of the problem reported in the ML are 
related to external disk used in USB enclousure, which most of the time lie 
about this aspect.

GB



-- 
gpg @keyserver.linux.it: Goffredo Baroncelli 
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Chris Murphy]

On Thu, Feb 11, 2016 at 7:58 AM, Austin S. Hemmelgarn
 wrote:
> On 2016-02-11 09:14, Goffredo Baroncelli wrote:
>>
>> On 2016-02-10 20:59, Austin S. Hemmelgarn wrote:
>> [...]
>>>
>>> Again, a torn write to the metadata referencing the block (stripe in
>>> this case I believe) will result in loosing anything written by the
>>> update to the stripe.
>>
>>
>> I think that the order matters: first the data block are written (in a new
>> location, so the old data are untouched), then the metadata, from the leafs
>> up to the upper node (again in a new location), then the superblock which
>> references to the upper node of the tree(s).
>>
>> If you interrupt the writes in any time, the filesystem can survive
>> because the old superblock-metadata-tree and data-block are still valid
>> until the last pieces (the new superblock) is written.
>>
>> And if this last step fails, the checksum shows that the super-block is
>> invalid and the old one is taken in consideration.
>
> You're not understanding what I'm saying.  If a write fails anywhere during
> the process of updating the metadata, up to and including the super-block,
> then you loose the data writes that triggered the metadata update.  This
> doesn't result in a broken filesystem, but it does result in data loss, even
> if it's not what most people think of as data loss.
>
> To make a really simplified example, assume we have a single block of data
> (D) referenced by a single metadata block (M) and a single super-block
> referencing the metadata block (S).  On a COW filesystem, when you write to
> D, it allocates and writes a new block (D2) to store the data, then
> allocates and writes a new metadata block (M2) to point to D2, and then
> updates the superblock in-place to point to M2.  If the write to M2 fails,
> you loose all new data in D2 that wasn't already in D.  There is no way that
> a COW filesystem can avoid this type of data loss without being able to
> force the underlying storage to atomically write out all of D2, M2, and S at
> the same time, it's an inherent issue in COW semantics in general, not just
> filesystems.

Sure but this is not a write hole. This exact same problem happens on
a single device file system. Do you know if raid56 parity strips are
considered part of (D) or (M)? In any case I think the Btrfs write
hole is different than what you're talking about.

The concern about the parity raid write hole is data is written OK,
and only in the event a device goes missing or there's an IO error on
a data containing block, such that reconstruction from parity is
required, if there's a bad or torn write for parity then the
reconstruction is bad and you won't know it. That's the key thing:
silent corruption during reconstruction.

Of all the problems we're having with raid56 in the general sense, let
alone the Btrfs specific case, the raid56 write hole seems like an
astronomically minor issue. What I'm much more curious about is how
these stripes are even being COWd in the first place, and how many
more IOs we're hit with compared to the same transaction on a single
device.




>>

>> The only critical thing, is that the hardware has to not lie about the
>> fact that the data reached the platter. Most of the problem reported in the
>> ML are related to external disk used in USB enclousure, which most of the
>> time lie about this aspect.
>
> That really depends on what you mean by 'lie about the data being on the
> platter'.  All modern hard disks have a write cache, and a decent percentage
> don't properly support flushing the write cache except by waiting for it to
> drain, many of them arbitrarily re-order writes within the cache, and none
> that I've seen have a non-volatile write cache, and therefore all such disks
> arguably lie about when the write is actually complete.  SSD's add yet
> another layer of complexity to this, because the good ones have either a
> non-volatile write cache, or have built-in batteries or super-capacitors to
> make sure they can flush the write cache when power is lost, so some SSD's
> can behave just like HDD's do and claim the write is complete when it hits
> the cache without technically lying, but most SSD's don't document whether
> they do this or not.

Yeah I think the ship of knowing what happens inside these boxes is in
the process of sailing away, if not gone. On the plus side, they do
all of this faster than a HDD would. So there's a better chance of the
command queue in the write cache actually completing to stable media
than is the case for a HDD *IF* the manufacturer has done the work and
testing.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Austin S. Hemmelgarn]

On 2016-02-09 15:39, Chris Murphy wrote:

On Fri, Feb 5, 2016 at 12:36 PM, Mackenzie Meyer  wrote:



RAID 6 write holes?


I don't even understand the nature of the write hole on Btrfs. If
modification is still always COW, then either an fs block, a strip, or
whole stripe write happens, I'm not sure where the hole comes from. It
suggests some raid56 writes are not atomic.
It's an issue of torn writes in this case, not of atomicity of BTRFS. 
Disks can't atomically write more than sector size chunks, which means 
that almost all BTRFS filesystems are doing writes that disks can't 
atomically complete.  Add to that that we serialized writes to different 
devices, and it becomes trivial to lose some data if the system crashes 
while BTRFS is writing out a stripe (it shouldn't screw up existing data 
though, you'll just loose whatever you were trying to write).


One way to minimize this which would also boost performance on slow 
storage would be to avoid writing parts of the stripe that aren't 
changed (so for example, if only one disk in the stripe actually has 
changed data, only write that and the parities).


If you're worried about raid56 write holes, then a.) you need a server
running this raid where power failures or crashes don't happen b.)
don't use raid56 c.) use ZFS.
It's not just BTRFS that has this issue though, ZFS does too, it just 
recovers more gracefully than BTRFS does, and even with the journaled 
RAID{5,6} support that's being added in MDRAID (and by extension DM-RAID 
and therefore LVM), it still has the same issue, it just moves it 
elsewhere (in this case, it has problems if there's a torn write to the 
journal).



RAID 6 stability?
Any articles I've tried looking for online seem to be from early 2014,
I can't find anything recent discussing the stability of RAID 5 or 6.
Are there or have there recently been any data corruption bugs which
impact RAID 6? Would you consider RAID 6 safe/stable enough for
production use?


It's not stable for your use case, if you have to ask others if it's
stable enough for your use case. Simple as that. Right now some raid6
users are experiencing remarkably slow balances, on the order of
weeks. If device replacement rebuild times are that long, I'd say it's
disqualifying for most any use case, just because there are
alternatives that have better fail over behavior than this. So far
there's no word from any developers what the problem might be, or
where to gather more information. So chances are they're already aware
of it but haven't reproduced it, or isolated it, or have a fix for it
yet.
Double on this, we should probably put something similar on the wiki, 
and this really applies to any feature, not just raid56.



Do you still strongly recommend backups, or has stability reached a
point where backups aren't as critical? I'm thinking from a data
consistency standpoint, not a hardware failure standpoint.


You can't separate them. On completely stable hardware, stem to stern,
you'd have no backups, no Btrfs or ZFS, you'd just run linear/concat
arrays with XFS, for example. So you can't just hand wave the hardware
part away. There are bugs in the entire storage stack, there are
connectors that can become intermittent, the system could crash. All
of these affect data consistency.
I may be wrong, but I believe the intent of this question was to try and 
figure out how likely BTRFS itself is to cause crashes or data 
corruption, independent of the hardware. In other words, 'Do I need to 
worry significantly about BTRFS in planning for disaster recovery, or 
can I focus primarily on the hardware itself?' or 'Is the most likely 
failure mode going to be hardware failure, or software?'. In general, 
right now I'd say that using BTRFS in traditional multi-device setup 
(nothing more than raid1 or possibly raid10), you've got roughly a 50% 
chance of an arbitrary crash being a software issue instead of hardware. 
Single disk, I'd say it's probably closer to 25%, and raid56 I'd say 
it's probably closer to 75%. By comparison, I'd say that with ZFS it's 
maybe a 5% chance (ZFS is developed as enterprise level software, it has 
to work, period), and with XFS on LVM raid, probably about 15% (similar 
to ZFS, XFS is supposed to be enterprise level software, the difference 
here comes from LVM, which has had some interesting issues recently due 
to incomplete testing of certain things before they got pushed upstream).


Stability has not reach a point where backups aren't as critical. I
don't really even know what that means though. No matter Btrfs or not,
you need to be doing backups such that if the primary stack is a 100%
loss without notice, is not a disaster. Plan on having to use it. If
you don't like the sound of that, look elsewhere.
What your using has impact on how you need to do backups.  For someone 
who can afford long periods of down time for example, it may be 
perfectly fine to use something like Amazon S3 Glacier storage 

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Chris Murphy]

On Wed, Feb 10, 2016 at 6:57 AM, Austin S. Hemmelgarn
 wrote:

> It's an issue of torn writes in this case, not of atomicity of BTRFS. Disks
> can't atomically write more than sector size chunks, which means that almost
> all BTRFS filesystems are doing writes that disks can't atomically complete.
> Add to that that we serialized writes to different devices, and it becomes
> trivial to lose some data if the system crashes while BTRFS is writing out a
> stripe (it shouldn't screw up existing data though, you'll just loose
> whatever you were trying to write).

I follow all of this. I still don't know how a torn write leads to a
write hole in the conventional sense though. If the write is partial,
a pointer never should have been written to that unfinished write. So
the pointer that's there after a crash should either point to the old
stripe or new stripe (which includes parity), not to the new data
strips but an old (stale) parity strip for that partial stripe write
that was interrupted. It's easy to see how conventional raid gets this
wrong because it has no pointers to strips, those locations are known
due to the geometry (raid level, layout, number of devices) and fixed.
I don't know what rmw looks like on Btrfs raid56 without overwriting
the stripe - a whole new cow'd stripe, and then metadata is updated to
reflect the new location of that stripe?




> One way to minimize this which would also boost performance on slow storage
> would be to avoid writing parts of the stripe that aren't changed (so for
> example, if only one disk in the stripe actually has changed data, only
> write that and the parities).

I'm pretty sure that's part of rmw, which is not a full stripe write.
At least there appears to be some distinction in raid56.c between
them. The additional optimization that md raid has had for some time
is the ability during rmw of a single data chunk (what they call
strips, or the smallest unit in a stripe), they can actually optimize
the change down to a sector write. So they aren't even doing full
chunk/strip writes either. The parity strip though I think must be
completely rewritten.


>>
>>
>> If you're worried about raid56 write holes, then a.) you need a server
>> running this raid where power failures or crashes don't happen b.)
>> don't use raid56 c.) use ZFS.
>
> It's not just BTRFS that has this issue though, ZFS does too,

Well it's widely considered to not have the write hole. From a ZFS
conference I got this tidbit on how they closed the write hole, but I
still don't understand why they'd be pointing to a partial (torn)
write in the first place:

"key insight was realizing instead of treating a stripe as it's a
"stripe of separate blocks" you can take a block and break it up into
many sectors and have a stripe across the sectors that is of one logic
block, that eliminates the write hole because even if the write is
partial until all of those writes are complete there's not going to be
an uber block referencing any of that." –Bonwick
https://www.youtube.com/watch?v=dcV2PaMTAJ4
14:45


> What your using has impact on how you need to do backups.  For someone who
> can afford long periods of down time for example, it may be perfectly fine
> to use something like Amazon S3 Glacier storage (which has a 4 hour lead
> time on restoration for read access) for backups. OTOH, if you can't afford
> more than a few minutes of down time and want to use BTRFS, you should
> probably have full on-line on-site backups which you can switch in on a
> moments notice while you fix things.

Right or use glusterfs or ceph if you need to stay up and running
during a total brick implosion. Quite honestly, I would much rather
see Btrfs single support multiple streams per device, like XFS does
with allocation groups when used on linear/concat of multiple devices;
two to four per



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Austin S. Hemmelgarn]

On 2016-02-10 14:06, Chris Murphy wrote:

On Wed, Feb 10, 2016 at 6:57 AM, Austin S. Hemmelgarn
 wrote:


It's an issue of torn writes in this case, not of atomicity of BTRFS. Disks
can't atomically write more than sector size chunks, which means that almost
all BTRFS filesystems are doing writes that disks can't atomically complete.
Add to that that we serialized writes to different devices, and it becomes
trivial to lose some data if the system crashes while BTRFS is writing out a
stripe (it shouldn't screw up existing data though, you'll just loose
whatever you were trying to write).


I follow all of this. I still don't know how a torn write leads to a
write hole in the conventional sense though. If the write is partial,
a pointer never should have been written to that unfinished write. So
the pointer that's there after a crash should either point to the old
stripe or new stripe (which includes parity), not to the new data
strips but an old (stale) parity strip for that partial stripe write
that was interrupted. It's easy to see how conventional raid gets this
wrong because it has no pointers to strips, those locations are known
due to the geometry (raid level, layout, number of devices) and fixed.
I don't know what rmw looks like on Btrfs raid56 without overwriting
the stripe - a whole new cow'd stripe, and then metadata is updated to
reflect the new location of that stripe?

I agree, it's not technically a write hole in the conventional sense, 
but the terminology has become commonplace for data loss in RAID{5,6} 
due to a failure somewhere in the write path, and this does fit in that 
sense.  In this case the failure is in writing out the metadata that 
references the blocks instead of in writing out the blocks themselves. 
Even though you don't loose any existing data, you still loose anything 
that you were trying to write out.





One way to minimize this which would also boost performance on slow storage
would be to avoid writing parts of the stripe that aren't changed (so for
example, if only one disk in the stripe actually has changed data, only
write that and the parities).


I'm pretty sure that's part of rmw, which is not a full stripe write.
At least there appears to be some distinction in raid56.c between
them. The additional optimization that md raid has had for some time
is the ability during rmw of a single data chunk (what they call
strips, or the smallest unit in a stripe), they can actually optimize
the change down to a sector write. So they aren't even doing full
chunk/strip writes either. The parity strip though I think must be
completely rewritten.
I actually wasn't aware that BTRFS did this (it's been a while since I 
looked at the kernel code), although I'm glad to hear it does.






If you're worried about raid56 write holes, then a.) you need a server
running this raid where power failures or crashes don't happen b.)
don't use raid56 c.) use ZFS.


It's not just BTRFS that has this issue though, ZFS does too,


Well it's widely considered to not have the write hole. From a ZFS
conference I got this tidbit on how they closed the write hole, but I
still don't understand why they'd be pointing to a partial (torn)
write in the first place:

"key insight was realizing instead of treating a stripe as it's a
"stripe of separate blocks" you can take a block and break it up into
many sectors and have a stripe across the sectors that is of one logic
block, that eliminates the write hole because even if the write is
partial until all of those writes are complete there's not going to be
an uber block referencing any of that." –Bonwick
https://www.youtube.com/watch?v=dcV2PaMTAJ4
14:45
Again, a torn write to the metadata referencing the block (stripe in 
this case I believe) will result in loosing anything written by the 
update to the stripe.  There is no way that _any_ system can avoid this 
issue without having the ability to truly atomically write out the 
entire metadata tree after the block (stripe) update.  Doing so would 
require a degree of tight hardware level integration that's functionally 
impossible for any general purpose system (in essence, the filesystem 
would have to be implemented in the hardware, not software).




What your using has impact on how you need to do backups.  For someone who
can afford long periods of down time for example, it may be perfectly fine
to use something like Amazon S3 Glacier storage (which has a 4 hour lead
time on restoration for read access) for backups. OTOH, if you can't afford
more than a few minutes of down time and want to use BTRFS, you should
probably have full on-line on-site backups which you can switch in on a
moments notice while you fix things.


Right or use glusterfs or ceph if you need to stay up and running
during a total brick implosion. Quite honestly, I would much rather
see Btrfs single support multiple streams per device, like XFS does
with allocation groups when used on linear/concat of 

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Psalle]

On 05/02/16 20:36, Mackenzie Meyer wrote:

RAID 6 stability?
I'll say more: currently, btrfs is in a state of flux where if you don't 
have a very recent kernel that's the first recommendation you're going 
to receive in case of problems. This means going out of stable packages 
in most distros.


Once you're in the bleeding kernel edge, you are obviously more likely 
to run into undiscovered bugs. I even see here people that has to patch 
the kernel with still non-mainline patches when trying to recover.


So don't for anything but testing.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Chris Murphy]

On Fri, Feb 5, 2016 at 12:36 PM, Mackenzie Meyer  wrote:

>
> RAID 6 write holes?

I don't even understand the nature of the write hole on Btrfs. If
modification is still always COW, then either an fs block, a strip, or
whole stripe write happens, I'm not sure where the hole comes from. It
suggests some raid56 writes are not atomic.

If you're worried about raid56 write holes, then a.) you need a server
running this raid where power failures or crashes don't happen b.)
don't use raid56 c.) use ZFS.


> RAID 6 stability?
> Any articles I've tried looking for online seem to be from early 2014,
> I can't find anything recent discussing the stability of RAID 5 or 6.
> Are there or have there recently been any data corruption bugs which
> impact RAID 6? Would you consider RAID 6 safe/stable enough for
> production use?

It's not stable for your use case, if you have to ask others if it's
stable enough for your use case. Simple as that. Right now some raid6
users are experiencing remarkably slow balances, on the order of
weeks. If device replacement rebuild times are that long, I'd say it's
disqualifying for most any use case, just because there are
alternatives that have better fail over behavior than this. So far
there's no word from any developers what the problem might be, or
where to gather more information. So chances are they're already aware
of it but haven't reproduced it, or isolated it, or have a fix for it
yet.

If you're prepared to make Btrfs better in the event you have a
problem, with possibly some delay in getting that volume up and
running again (including the likelihood of having to rebuild it from a
backup), then it might be compatible with your use case.

> Do you still strongly recommend backups, or has stability reached a
> point where backups aren't as critical? I'm thinking from a data
> consistency standpoint, not a hardware failure standpoint.

You can't separate them. On completely stable hardware, stem to stern,
you'd have no backups, no Btrfs or ZFS, you'd just run linear/concat
arrays with XFS, for example. So you can't just hand wave the hardware
part away. There are bugs in the entire storage stack, there are
connectors that can become intermittent, the system could crash. All
of these affect data consistency.

Stability has not reach a point where backups aren't as critical. I
don't really even know what that means though. No matter Btrfs or not,
you need to be doing backups such that if the primary stack is a 100%
loss without notice, is not a disaster. Plan on having to use it. If
you don't like the sound of that, look elsewhere.


> I plan to start with a small array and add disks over time. That said,
> currently I have mostly 2TB disks and some 3TB disks. If I replace all
> 2TB disks with 3TB disks, would BTRFS then start utilizing the full
> 3TB capacity of each disk, or would I need to destroy and rebuild my
> array to benefit from the larger disks?

Btrfs, or LVM raid, or  mdraid, and ZFS all let you grow arrays, each
has different levels of ease of doing this and how long it will take,
without having to recreate the file system from scratch.


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Psalle]

On 05/02/16 20:36, Mackenzie Meyer wrote:

Hello,

I've tried checking around on google but can't find information
regarding the RAM requirements of BTRFS and most of the topics on
stability seem quite old.


To keep my answer short: every time I've tried (offline) deduplication 
or raid5 pools I've ended with borked filesystems. Last attempt was 
about a year ago. Given that the pages you mention looked the same by 
then, I'd stay away of raid56 for anything but testing purposes. I 
haven't read anything about raid5 that increases my confidence in it 
recently (i.e. post 3.19 kernels). Dedup, OTOH, I don't know. What I 
used were third-party (I think?) things so the fault may have rested on 
them and not btrfs (does that makes sense?)


I'm building a new small raid5 pool as we speak, though, for throw-away 
data, so I hope to be favourably impressed.


Cheers.


So first would be memory requirements, my goal is to use deduplication
and compression. Approximately how many GB of RAM per TB of storage
would be recommended?

RAID 6 write holes?
The BTRFS wiki states that parity might be inconsistent after a crash.
That said, the wiki page for RAID 5/6 doesn't look like it has much
recent information on there. Has this issue been addressed and if not,
are there plans to address the RAID write hole issue? What would be a
recommended workaround to resolve inconsistent parity, should an
unexpected power down happen during write operations?

RAID 6 stability?
Any articles I've tried looking for online seem to be from early 2014,
I can't find anything recent discussing the stability of RAID 5 or 6.
Are there or have there recently been any data corruption bugs which
impact RAID 6? Would you consider RAID 6 safe/stable enough for
production use?

Do you still strongly recommend backups, or has stability reached a
point where backups aren't as critical? I'm thinking from a data
consistency standpoint, not a hardware failure standpoint.

I plan to start with a small array and add disks over time. That said,
currently I have mostly 2TB disks and some 3TB disks. If I replace all
2TB disks with 3TB disks, would BTRFS then start utilizing the full
3TB capacity of each disk, or would I need to destroy and rebuild my
array to benefit from the larger disks?


Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Duncan]

Mackenzie Meyer posted on Fri, 05 Feb 2016 14:36:33 -0500 as excerpted:

> Hello,
> 
> I've tried checking around on google but can't find information
> regarding the RAM requirements of BTRFS and most of the topics on
> stability seem quite old.
> 
> So first would be memory requirements, my goal is to use deduplication
> and compression. Approximately how many GB of RAM per TB of storage
> would be recommended?

The inline dedup patches are just reaching maturity and mainlining right 
now, and aren't in a release yet.  Dedup's not my use-case so I've not 
been following it /too/ closely, but briefly, as it's shaping up there's 
going to be two backends available, an on-device backend that will be a 
bit slower but should be more efficient at deduping, and an in-memory 
backend that will be fast but less efficient.  Dedup memory usage is 
configurable, however, with a rather low default of (IIRC) some MiB, 
certainly nothing like GiB, unless of course you configure it that way 
(which you may wish to if you have the memory and choose the memory-based 
backend).

The memory issue is thus neither dedup nor compression (which is on-the-
fly and uses very little additional memory).

Instead, btrfs memory issues tend to be scaling related, specifically, 
related to the number of subvolumes/snapshots, and to whether you have 
quotas activated.

I'm actually not sure what the current quota status is as of 4.4 and the 
in-development 4.5, but definitely, previous to that, quotas simply were 
not stable and had known-broken corner-cases as well as serious scaling 
issues, so the recommendation has been, you either need them or you 
don't: if you need them, use a filesystem where quotas are mature and 
stable; if not, use btrfs but leave quotas disabled as they have 
definitely caused many a user serious headaches as they simply didn't 
work well.  Unless of course you're specifically working with the devs to 
test current quota code, reporting results and potentially running 
debugging patches to get more info when there's problems, in which case 
go right ahead, as you're one of the folks that's helping to eventually 
make the feature stable and workable enough to actually depend on.

Btrfs snapshots/subvolumes are of course a relatively more stable btrfs 
feature and are regularly used by many.  But there remain scaling issues 
there as well, such that the recommended total filesystem cap on number 
of subvolumes (including snapshots, which are a special kind of subvolume 
that simply happens to share most of its data with other subvolumes) is 
no more than 1000-3000.  The number of snapshots of individual 
subvolumes, meanwhile, should be kept to 250 or so, which is actually 
pretty reasonable even if you're starting with for instance half-hourly 
auto-snapshotting (using snapper or the like), as long as a reasonable 
snapshot thinning schedule is followed as well.

(As I've posted several times, starting with half-hourly snapshots, 
thinning to hourly after say 12 hours, 2-hourly after 48 hours, daily 
after a week, and weekly after perhaps 13 weeks (a quarter), then keeping 
the remaining weeklys for another year, so 15 months of snapshots total, 
after which if you haven't backed up to other media you obviously aren't 
worried too much about losing the data anyway, you're still reasonably 
close to 250-300 snapshots per subvolume, thus allowing snapshotting of 
up to eight subvolumes on a similar program while still staying within 
the filesystem cap of 2000 or so snapshots/subvolumes.)

Because with btrfs snapshotting being so easy, what we too often see if 
people aren't warned about it, are people with 100K snapshots or the 
like.  And during normal runtime, other than perhaps some slowdown due to 
fragmentation, etc, even that works deceptively well.

Where the _problem_ occurs, however, is when you try to actually do 
filesystem maintenance on that monster!  Both btrfs balance and btrfs 
check slow down *dramatically*, to the point of practical unworkability, 
somewhere in the double-digit-K (tens of thousands) snapshots range.  
With 3000 it's noticeably slower than with only 1000, but while slow, 
it's still _usable_.  Which is why the 1000-3000 range.  Depending on 
people's pain point vs snapshotting needs, 1000 snapshots shouldn't be 
much of a problem yet, but might not be enough for people with more than 
3-4 subvolumes they want to keep snapshotted, while 3000 snapshots will 
already be hitting the pain point for many, but may be needed and still 
just fast enough to be tolerably usable given the tradeoff, for people 
with many subvolumes they want to keep snapshotted.

And based on reports, btrfs check with tens of thousands of snapshots is 
where memory usage goes thru the roof as well.  I'm not actually sure 
about balance in terms of memory usage, tho it's definitely much slower 
with that many snapshots.

It can also be noted that the problem affects snapshot deletion (unlike 
snapshot 

BTRFS RAM requirements, RAID 6 stability/write holes and expansion questions

[Mackenzie Meyer]

Hello,

I've tried checking around on google but can't find information
regarding the RAM requirements of BTRFS and most of the topics on
stability seem quite old.

So first would be memory requirements, my goal is to use deduplication
and compression. Approximately how many GB of RAM per TB of storage
would be recommended?

RAID 6 write holes?
The BTRFS wiki states that parity might be inconsistent after a crash.
That said, the wiki page for RAID 5/6 doesn't look like it has much
recent information on there. Has this issue been addressed and if not,
are there plans to address the RAID write hole issue? What would be a
recommended workaround to resolve inconsistent parity, should an
unexpected power down happen during write operations?

RAID 6 stability?
Any articles I've tried looking for online seem to be from early 2014,
I can't find anything recent discussing the stability of RAID 5 or 6.
Are there or have there recently been any data corruption bugs which
impact RAID 6? Would you consider RAID 6 safe/stable enough for
production use?

Do you still strongly recommend backups, or has stability reached a
point where backups aren't as critical? I'm thinking from a data
consistency standpoint, not a hardware failure standpoint.

I plan to start with a small array and add disks over time. That said,
currently I have mostly 2TB disks and some 3TB disks. If I replace all
2TB disks with 3TB disks, would BTRFS then start utilizing the full
3TB capacity of each disk, or would I need to destroy and rebuild my
array to benefit from the larger disks?


Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html