I try to figure out if I go into the right direction. I am a HUGE fan of
ZFS filesystem because if you use that filesystem with ECC memory and take
a JBOD of disks and put them in a
redundancy setup (ZFS3 or ZFS2) you have not only snapshots, compression
but also deduplication. The cache is hold in ZIL ssd/nvme to speed up file
access... however that
cache should always have redundancy as well ! The only point I do miss also
in ZFS is versioning....
So in my store I see a few issues:
1) it looks to me there is no versioning in s3ql either. This means that
feels very uncomfortable to me seen the new range of virii
2) the cache (this counts for s3ql as borgbackup as well) should always be
on a stable filesystem so if this is run on a single platter that is a very
weak point, seen that cache is synced in priority
Of course there is a way to mount readonly and use immutable trees, but
if it is mounted in a regular way, corrupt files (bitrot etc) are synced
upstream
3) there is no parity on the s3ql_data_* files , I like to see some error
detection and correction there in the form of s3ql_data_NNNNN.par2 (for
each data block)
and additional s3qlcmd to verify and if needed repair on that level in
a worse case scenario...
Compression saves me some space, deduplication not for the most part... I
think the latter is only usefull in big corporations with potential a lot
of duplicated files (blocks)
Both take a toll on the cpu as well.
For me encryption is mandatory and I am a huge fan of asymmetric encryption
in favor of symmetric encryption seen in the first case you use the public
key.
So I still need to find the right approach in this but seen there is no
versioning.....I have a problem.
I can not snapshot every hour the FS
Op vrijdag 10 september 2021 om 23:46:09 UTC+2 schreef dgas...@gmail.com:
>
>
> On Fri, Sep 10, 2021 at 2:37 PM David Gasaway <da...@gasaway.org> wrote:
>
>>
>>
>> On Thu, Sep 9, 2021 at 2:57 AM Amos T <amtr...@gmail.com> wrote:
>>
>>
>>> So in that case my a safely assume when borgbackup checks for data
>>> modification, the
>>> file in question is not downloaded ? Because in that case, my backup
>>> will generate a lot of traffic...
>>>
>>
>> The documentation for `borg create` explains the methodologies for
>> detecting changed files. Basically, with the defaults, it caches the
>> metadata of the files it backed up, into a local borg cache. It doesn't
>> even need to read anything at all from the destination (s3ql) file system
>> so long as the cache is intact. If this cache is lost or corrupt, borg has
>> to read (download, in the case of s3ql) the entire repository to rebuild
>> the cache.
>>
>
> I may have to take some of that back. It sounds like borg does keep
> components of the cache on the remote file system. So a local cache
> rebuild doesn't need to download the whole repository. `borg check`
> operations frequently do, however. The point I was trying to make is,
> under normal circumstances, only the local cache is used for change
> detection.
>
> --
> -:-:- David K. Gasaway
> -:-:- Email: da...@gasaway.org
>
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/s3ql/0eaa4d33-5653-40db-a113-d54c09841da0n%40googlegroups.com.
