I've being running newer version of just-backup-btrfs, which was configured to remove snapshots in batches ~ at least 3x100 at once (this is what I typically have in 1.5-2 days).
Snapshots transferring become much faster, however when I delete 300 snapshots
at once, well... you can imagine what happens, but I can afford this on desktop.
Seekwatcher fails to run on my system with following error:
~> sudo seekwatcher -t find.trace -o find.png -p 'find /backup_hdd > /dev/null'
-d /dev/sda1
Traceback (most recent call last):
File "/usr/bin/seekwatcher", line 58, in <module>
from seekwatcher import rundata
File "numpy.pxd", line 43, in seekwatcher.rundata (seekwatcher/rundata.c:7885)
ValueError: numpy.dtype does not appear to be the correct type object
I have no idea what does it mean, but generally I think if seeking because of
fragmentation is a real cause of performance degradation, then this is
something that BTRFS can improve, since I still have 65% of free space on BTRFS
partition that receives snapshots and fragmentation in this case seems weird.
P.S. I've unsubscribed from mailing list, cc me on answers, please.
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
Skype: nazar-pc
Diaspora: naza...@diaspora.mokrynskyi.com
Tox:
A9D95C9AA5F7A3ED75D83D0292E22ACE84BA40E912185939414475AF28FD2B2A5C8EF5261249
On 18.03.16 16:22, Nazar Mokrynskyi wrote:
>> But seriously, what are you doing that you can't lose more than 15
>> minutes of? Couldn't it be even 20 minutes, or a half-hour or an hour
>> with 15 minute snapshots only on the ssds (yes, I know the raid0 factor,
>> but the question still applies)?
> This is artificial psychological limit. Sometimes when you're actively
> coding, it is quite sad to loose even 5 minutes of work, since productivity
> is not constant. This is why 15 minutes was chosen as something that is not
> too critical. There is no other real reason behind this limit other than how
> I feel it.
>
>> And perhaps more importantly for your data, btrfs is still considered
>> "stabilizing, not fully stable and mature". Use without backups is
>> highly discouraged, but I'd suggest that btrfs in its current state might
>> not be what you're looking for if you can't deal with loss of more than
>> 15 minutes worth of changes anyway.
>>
>> Be that as it may...
>>
>> Btrfs is definitely not yet optimized. In many cases it reads or writes
>> only one device at a time, for instance, even in RaidN configuration.
>> And there are definitely snapshot scaling issues altho at your newer 500
>> snapshots total that shouldn't be /too/ bad.
> As an (relatively) early adopter I'm fine using experimental stuff with extra
> safeties like backups (hey, I've used it even without those while back:)). I
> fully acknowledge what is current state of BTRFS and want to help make it
> even better by stressing issues that me and other users encounter, searching
> for solutions, etc.
>
>> Dealing with reality, regardless of how or why, you currently have a
>> situation of intolerably slow receives that needs addressed. From a
>> practical perspective you said an ssd for backups is ridiculous and I
>> can't disagree, but there's another "throw hardware at it" solution that
>> might be a bit more reasonable...
>>
>> Spinning rust hard drives are cheap. What about getting another one, and
>> alternating your backup receives between them? That would halve the load
>> to one every thirty minutes, without changing your 15-minute snapshot and
>> backup policy at all. =:^)
>>
>> So that gives you two choices for halving the load to the spinning rust.
>> Either decide you really can live with half-hour loss of data, or throw
>> only a relatively small amount of money (well, as long as you have room
>> to plug in another sata device anyway, otherwise...) at it for a second
>> backup device, and alternate between them.
> Yes, I'm leaning toward earning new hardware right now, fortunately, laptop
> allows me to insert 2 x mSATA + 2 x 2.5 SATA drives, so I have exactly 2.5
> SATA slot free.
>
>> OTOH, since you mentioned possible coding, optimization might not be a
>> bad thing, if you're willing to put in the time necessary to get up to
>> speed with the code and can work with the other devs in terms of timing,
>> etc. But that will definitely take significant time even if you do it,
>> and the alternating backup solution can be put to use as soon as you can
>> get another device plugged in and setup. =:^)
> I'm not coding C/C++, so my capabilities to improve BTRFS itself are limited,
> but I'm always trying to find the reason and fix it instead of living with
> workarounds forever.
>
> I'll play with Seekwatcher and optimizing snapshots deletion and will post an
> update afterwards.
>
> Sincerely, Nazar Mokrynskyi
> github.com/nazar-pc
> Skype: nazar-pc
> Diaspora: naza...@diaspora.mokrynskyi.com
> Tox:
> A9D95C9AA5F7A3ED75D83D0292E22ACE84BA40E912185939414475AF28FD2B2A5C8EF5261249
>
smime.p7s
Description: Кріптографічний підпис S/MIME
