Re: [BackupPC-users] btrfs questions

[Ghislain Adnet]

Le 06/03/2021 à 15:24, Paul Leyland a écrit :

I've seen bit-rot on a few disks out of hundreds used over the last
35-ish years.

I am now storing /var/lib/backuppc on a ZFS RAID since the last
catastophic disk failure. Sure enough one of those disks started writing
garbage and then was taken off-line through infant mortality. The pool
kept going. A year or so later a different disk went off-line, with a
dying SATA cable this time. The pool kept going. In both cases
rebuilding the array ("re-silvering") happened automagically.

Very happy with ZFS myself. YMMV.


Curious i have ZFS on my backuppc4 machines and i got horrible performances.

/usr/share/backuppc/bin/BackupPC_refCountUpdate

is completly killing the machine for hours while on btrfs and ext4 it does not 
cause any issue.

i am planning to move out of ZFS as soon as i can because notaime, nosync and 
all optimisation like l2arc on ssd have not helped the problem :(

--
cordialement,
Ghislain



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Alexander Moisseev via BackupPC-users]

On 06.03.2021 19:28, Paul Leyland wrote:

But backuppc works just fine on a BSD-licensed mainline kernel.

On 06/03/2021 14:46, Richard Shaw wrote:

On Sat, Mar 6, 2021 at 8:26 AM Paul Leyland mailto:paul.leyl...@gmail.com>> wrote:


Very happy with ZFS myself. YMMV.


If only they would move to a FOSS license instead of CDDL it could be included 
in the mainline kernel.


Moreover, BackupPC works fine on FreeBSD which has ZFS support from the kernel 
for years.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Michael Stowe]

On 2021-03-06 01:55, John Botha (SourceForge) wrote:


Hi,

I am about to take the plunge with BackupPC, and would appreciate input 
on the following three points regarding using btrfs for the data (bit 
rot protection is key for me).


From what I've read, btrfs (like many file systems) suffers over time 
when fragmentation increases. I have seen suggestions such as not to 
put data bases on btrfs because of this, but that just seems silly at a 
number of levels. At the least one should take special care with data 
bases (know your data and adjust your maintenance accordingly), but 
that holds for any data base on any FS. My question is how best to 
approach this with a combination of rebalancing and scrubbing, or if 
there is another way or other aspects to keep in mind.


I won't be using snapshots in btrfs since BackupPC effectively 
implements its own. When I read up how btrfs implements COW I thought 
it would be safest to use nodatacow, but then read that doing so would 
also stop bit rot protection, so that's a real bummer. Am I missing 
something, or do I have that right?


Given how I understand BackupPC implements compression, I'd rather have 
btrfs handle de/compression, as that would seem to involve less time 
doing redundant calculations. Does that make sense?


Thanks in advance for constructive input, as I have seen some flame 
wars around the use of btrfs.


8-)
John


It all depends on what you mean by "suffers," but the BackupPC pattern 
does not update individual files, and therefore does not cause the kind 
of fragmentation that causes thrashing.  To be succinct, it's not much 
to worry about.  I do scrub occasionally (every 6-12 months) but I don't 
bother rebalancing unless the physical shape of the array changes.  For 
example after replacing a failed drive or adding capacity.  It's not a 
bad idea to monitor logs or btrfs itself for errors which can mean early 
signs of drive failure, but presumably you'd also monitor the physical 
devices.


What you are missing is that COW doesn't really happen with BackupPC, so 
I wouldn't recommend taking any special precautions to turn it off.  (In 
fact, I'd go so far as to say don't ever do this unless you have a very 
good reason, and I'd seriously question that reason, because most of the 
reasons I've heard for doing so are based on magical thinking and 
nonsense.)


In practice, it doesn't matter much whether BackupPC implements 
compression or btrfs does.  Don't do both, obviously, because that's 
pointless.  Because BackupPC has it on by default and BTRFS doesn't, I 
went that way since further tweaking is not going to represent 
significant incremental gains, but you may find doing so worthwhile.___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Paul Leyland]

But backuppc works just fine on a BSD-licensed mainline kernel.

On 06/03/2021 14:46, Richard Shaw wrote:
> On Sat, Mar 6, 2021 at 8:26 AM Paul Leyland  > wrote:
>
>
> Very happy with ZFS myself. YMMV.
>
>
> If only they would move to a FOSS license instead of CDDL it could be
> included in the mainline kernel.
>
> Thanks,
> Richard 
>
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:https://github.com/backuppc/backuppc/wiki
> Project: https://backuppc.github.io/backuppc/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Richard Shaw]

Standard disclaimer, I'm not a btrfs expert...

On Sat, Mar 6, 2021 at 6:52 AM John Botha (SourceForge) <
sourcefo...@yellowacorn.com> wrote:

> From what I've read, btrfs (like many file systems) suffers over time when
> fragmentation increases. I have seen suggestions such as not to put data
> bases on btrfs because of this, but that just seems silly at a number of
> levels. At the least one should take special care with data bases (know
> your data and adjust your maintenance accordingly), but that holds for any
> data base on any FS. My question is how best to approach this with a
> combination of rebalancing and scrubbing, or if there is another way or
> other aspects to keep in mind.
>

Defragmenting is easy enough... But yes, large files that change are
generally not a good idea on a COW filesystem as it causes excessive
writes. Since you mention rebalancing I assume you mean some sort of disk
array?



> I won't be using snapshots in btrfs since BackupPC effectively implements
> its own. When I read up how btrfs implements COW I thought it would be
> safest to use nodatacow, but then read that doing so would also stop bit
> rot protection, so that's a real bummer. Am I missing something, or do I
> have that right?
>

I don't know about stop, but you would be losing one of the major benefits
of btrfs since it writes the file before "erasing" the old one. I think
btrfs is very good for system and /home drives unless you're working with
large files (databases, major video editing, etc). It even has some
advantages over typical RAID configurations as long as you don't have buggy
drive firmware and don't mind scrubbing your file system on a regular
basis. I'm not sure it's a great candidate for BackupPC at this time.

Here's a thread on the Fedora mailing list when I was looking into btrfs
RAID (not for BackupPC)

https://lists.fedoraproject.org/archives/list/us...@lists.fedoraproject.org/message/FNK3XEPHAMN4L4DEYQFMEJPA2OQVW3AM/

Chris is very helpful and there's a lot of good info in the thread.


Given how I understand BackupPC implements compression, I'd rather have
> btrfs handle de/compression, as that would seem to involve less time doing
> redundant calculations. Does that make sense?
>

I think the only real advantage here is that the btrfs compression would be
multi-threaded, I don't think BackupPC is but someone please correct me if
I'm wrong. I run BackupPC on a dedicated server so I'm not really worried
how long it takes as long as it can keep up. Which for my home use it does.

Thanks,
Richard
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Richard Shaw]

On Sat, Mar 6, 2021 at 8:26 AM Paul Leyland  wrote:

>
> Very happy with ZFS myself. YMMV.
>

If only they would move to a FOSS license instead of CDDL it could be
included in the mainline kernel.

Thanks,
Richard
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[Paul Leyland]

I've seen bit-rot on a few disks out of hundreds used over the last
35-ish years.

I am now storing /var/lib/backuppc on a ZFS RAID since the last
catastophic disk failure. Sure enough one of those disks started writing
garbage and then was taken off-line through infant mortality. The pool
kept going. A year or so later a different disk went off-line, with a
dying SATA cable this time. The pool kept going. In both cases
rebuilding the array ("re-silvering") happened automagically.

Very happy with ZFS myself. YMMV.

Paul

On 06/03/2021 13:50, G.W. Haywood via BackupPC-users wrote:
> Hi there,
>
> On Sat, 6 Mar 2021, John Botha (SourceForge) wrote:
>
>> ... take the plunge with BackupPC, ... bit rot protection is key ...
>> ...
>> ... fragmentation ... how best to approach this with a combination
>> of rebalancing and scrubbing, or if there is another way or other
>> aspects to keep in mind.
>> ...
>> ... I thought it would be safest to use nodatacow, but then read
>> that doing so would also stop bit rot protection, so that's a real
>> bummer. Am I missing something, or do I have that right?
>> ...
>> ... have btrfs handle de/compression, as that would seem to involve
>> less time doing redundant calculations. Does that make sense?
>> ...
>> ... seen some flame wars around the use of btrfs.
>
> I don't want to add fuel to any flames.
>
> In my view you're making it more difficult for yourself than you need
> to (or indeed should do) if you're just starting out with BackupPC.
>
> My take on it is that you will have quite enough on your plate getting
> BackupPC bedded down - so it's doing what you want in your particular
> circumstances, and you're comfortable with that - without adding into
> the mix a whole bunch of variables which don't need to be variables.
>
> If 'bit rot' protection is key to you, then set up BackupPC to avoid
> any possibility of it happening, spend a few months (or perhaps years)
> making sure that it isn't happening, and worry about filesystem(s),
> and any quirks they may have, some other time.
>
> I personally have never seen any evidence of what I imagine might be
> called 'bit rot' because in my view if something like that's happening
> then the system is badly broken and it needs fixing.  But I have seen
> plenty of damaged filesystems.  When I have had experience of damaged
> filesystems, I believe it's fair to say that the newer the filesystem,
> the more difficult it has been to repair it.  The first (and last!!!)
> ReiserFS I ever used failed catastrophically and was never recovered.
> I've recovered everything from DOS to EXT/2/3/4 systems, usually with
> little difficulty; I've never used BTRFS so I can't offer any comment
> on its repairability.
>
> Right now I use EXT4 almost exclusively, and there would have to be a
> really technologically disruptive development in filesystem capability
> (like an order of magnitude improvement in some performance metric) to
> encourage me even to consider changing to anything else.  I don't care
> if anybody thinks I'm an old stick-in-the-mud, I just want it to work.
>
> The other day when I was out with one of my dogs I fell into chatting
> with a couple of other walkers.  This particular dog is a difficult
> case from the rescue.  One of the walkers said "you seem to have a
> calm aura about you".  Of course that's necessary for these difficult
> rescue cases.  I thanked her for the compliment although I didn't say
> "it's because I use BackupPC and EXT4" - which wouldn't have been too
> far from the truth.
>


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] btrfs questions

[G.W. Haywood via BackupPC-users]

Hi there,

On Sat, 6 Mar 2021, John Botha (SourceForge) wrote:


... take the plunge with BackupPC, ... bit rot protection is key ...
...
... fragmentation ... how best to approach this with a combination
of rebalancing and scrubbing, or if there is another way or other
aspects to keep in mind.
...
... I thought it would be safest to use nodatacow, but then read
that doing so would also stop bit rot protection, so that's a real
bummer. Am I missing something, or do I have that right?
...
... have btrfs handle de/compression, as that would seem to involve
less time doing redundant calculations. Does that make sense?
...
... seen some flame wars around the use of btrfs.


I don't want to add fuel to any flames.

In my view you're making it more difficult for yourself than you need
to (or indeed should do) if you're just starting out with BackupPC.

My take on it is that you will have quite enough on your plate getting
BackupPC bedded down - so it's doing what you want in your particular
circumstances, and you're comfortable with that - without adding into
the mix a whole bunch of variables which don't need to be variables.

If 'bit rot' protection is key to you, then set up BackupPC to avoid
any possibility of it happening, spend a few months (or perhaps years)
making sure that it isn't happening, and worry about filesystem(s),
and any quirks they may have, some other time.

I personally have never seen any evidence of what I imagine might be
called 'bit rot' because in my view if something like that's happening
then the system is badly broken and it needs fixing.  But I have seen
plenty of damaged filesystems.  When I have had experience of damaged
filesystems, I believe it's fair to say that the newer the filesystem,
the more difficult it has been to repair it.  The first (and last!!!)
ReiserFS I ever used failed catastrophically and was never recovered.
I've recovered everything from DOS to EXT/2/3/4 systems, usually with
little difficulty; I've never used BTRFS so I can't offer any comment
on its repairability.

Right now I use EXT4 almost exclusively, and there would have to be a
really technologically disruptive development in filesystem capability
(like an order of magnitude improvement in some performance metric) to
encourage me even to consider changing to anything else.  I don't care
if anybody thinks I'm an old stick-in-the-mud, I just want it to work.

The other day when I was out with one of my dogs I fell into chatting
with a couple of other walkers.  This particular dog is a difficult
case from the rescue.  One of the walkers said "you seem to have a
calm aura about you".  Of course that's necessary for these difficult
rescue cases.  I thanked her for the compliment although I didn't say
"it's because I use BackupPC and EXT4" - which wouldn't have been too
far from the truth.

--

73,
Ged.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/