On Jul 11 2020, Ivan Shapovalov <inte...@intelfx.name> wrote:
> On 2020-07-10 at 19:54 +0100, Nikolaus Rath wrote:
>> On Jul 10 2020, Daniel Jagszent <dan...@jagszent.de> wrote:
>> > > Ah yes, compression and probably encryption will indeed preclude
>> > > any sort of partial block caching. An implementation will have to
>> > > be limited to plain uncompressed blocks, which is okay for my
>> > > use- case though (borg provides its own encryption and
>> > > compression anyway). [...]
>> > Compression and encryption are integral parts of S3QL and I would
>> > argue that disabling them is only an edge case.
>>
>> If I were to write S3QL from scratch, I would probably not support
>> this at all, right. However, since the feature is present, I think we
>> ought to consider it fully supported ("edge case" makes it sound as
>> if this isn't the case).
>>
>>
>> > I might be wrong but I think Nikolaus (maintainer of S3QL) will not
>> > accept such a huge change into S3QL that is only beneficial for an
>> > edge
>> > case.
>>
>> Never say never, but the bar is certainly high here. I think there
>> are
>> more promising avenues to explore - eg. storing the
>> compressed/uncompressed offset mapping to make partial retrieval work
>> for all cases.
>
> Hmm, I'm not sure how's that supposed to work.
>
> AFAICS, s3ql uses "solid compression", meaning that the entire block is
> compressed at once. It is generally impossible to extract a specific
> range of uncompressed data without decompressing the whole stream.[1]
At least bzip2 always works in blocks, IIRC blocks are at most 900 kB
(for highest compression settings). I wouldn't be surprised if the same
holds for LZMA.
We could track the size of each compressed block, and store it as part
of the metadata of the object (so it doesn't blow-up the SQLite table).
> Encryption does not pose this kind of existential problem — AES is used
> in CTR mode, which theoretically permits random-access decryption — but
> the crypto library in use, python-cryptography, doesn't seem to permit
> this sort of trickery.
Worst case you can feed X bytes of garbage into the decrypter and then
start with the partial block - with CTR you should get the right
output.
Best,
Nikolaus
--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/s3ql/87pn92nw27.fsf%40thinkpad.rath.org.
