On 2020-07-11 at 12:13 +0100, Nikolaus Rath wrote:
> On Jul 11 2020, Ivan Shapovalov <inte...@intelfx.name> wrote:
> > On 2020-07-10 at 19:54 +0100, Nikolaus Rath wrote:
> > > On Jul 10 2020, Daniel Jagszent <dan...@jagszent.de> wrote:
> > > > > Ah yes, compression and probably encryption will indeed
> > > > > preclude
> > > > > any sort of partial block caching. An implementation will
> > > > > have to
> > > > > be limited to plain uncompressed blocks, which is okay for my
> > > > > use- case though (borg provides its own encryption and
> > > > > compression anyway). [...]
> > > > Compression and encryption are integral parts of S3QL and I
> > > > would
> > > > argue that disabling them is only an edge case.
> > >
> > > If I were to write S3QL from scratch, I would probably not
> > > support
> > > this at all, right. However, since the feature is present, I
> > > think we
> > > ought to consider it fully supported ("edge case" makes it sound
> > > as
> > > if this isn't the case).
> > >
> > >
> > > > I might be wrong but I think Nikolaus (maintainer of S3QL) will
> > > > not
> > > > accept such a huge change into S3QL that is only beneficial for
> > > > an
> > > > edge
> > > > case.
> > >
> > > Never say never, but the bar is certainly high here. I think
> > > there
> > > are
> > > more promising avenues to explore - eg. storing the
> > > compressed/uncompressed offset mapping to make partial retrieval
> > > work
> > > for all cases.
> >
> > Hmm, I'm not sure how's that supposed to work.
> >
> > AFAICS, s3ql uses "solid compression", meaning that the entire
> > block is
> > compressed at once. It is generally impossible to extract a
> > specific
> > range of uncompressed data without decompressing the whole
> > stream.[1]
>
> At least bzip2 always works in blocks, IIRC blocks are at most 900 kB
> (for highest compression settings). I wouldn't be surprised if the
> same
> holds for LZMA.True, I forgot that bzip2 is inherently block-based. Not sure about LZMA or gzip, but there is still a significant obstacle: how would you extract this information from the compression libraries? > > We could track the size of each compressed block, and store it as > part > of the metadata of the object (so it doesn't blow-up the SQLite > table). > > > Encryption does not pose this kind of existential problem — AES is > > used > > in CTR mode, which theoretically permits random-access decryption — > > but > > the crypto library in use, python-cryptography, doesn't seem to > > permit > > this sort of trickery. > > Worst case you can feed X bytes of garbage into the decrypter and > then > start with the partial block - with CTR you should get the right > output. Yes, that could probably work. Still feels like a grand hack. -- Ivan Shapovalov / intelfx / -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/85251a7f8c1176770ca65acc679266d89c3a0211.camel%40intelfx.name.
signature.asc
Description: This is a digitally signed message part
