Re: [HACKERS] Custom compression methods

[Ildus K]

On Mon, 1 Jul 2019 at 17:28, Alexander Korotkov 
wrote:

> On Mon, Jul 1, 2019 at 5:51 PM Alvaro Herrera 
> wrote:
> > On 2019-Jul-01, Alexander Korotkov wrote:
> >
> > > As I get we're currently need to make high-level decision of whether
> > > we need this [1].  I was going to bring this topic up at last PGCon,
> > > but I didn't manage to attend.  Does it worth bothering Ildus with
> > > continuous rebasing assuming we don't have this high-level decision
> > > yet?
> >
> > I agree that having to constantly rebase a patch that doesn't get acted
> > upon is a bit pointless.  I see a bit of a process problem here: if the
> > patch doesn't apply, it gets punted out of commitfest and reviewers
> > don't look at it.  This means the discussion goes unseen and no
> > decisions are made.  My immediate suggestion is to rebase even if other
> > changes are needed.
>
> OK, let's do this assuming Ildus didn't give up yet :)
>

No, I still didn't give up :)
I'm going to post rebased version in few days. I found that are new
conflicts with
a slice decompression, not sure how to figure out them for now.

Also I was thinking maybe there is a point to add possibility to compress
any data
that goes to some column despite toast threshold size. In our company we
have
types that could benefit from compression even on smallest blocks.

Since pluggable storages were committed I think I should notice that
compression
methods also can be used by them and are not supposed to work only with
toast tables.
Basically it's just an interface to call compression functions which are
related with some column.

Best regards,
Ildus Kurbangaliev

Re: [HACKERS] Custom compression methods

[Ildus K]

On Tue, 21 Nov 2017 18:47:49 +0100
Tomas Vondra  wrote:

 
> 
> I propose to use either
> 
>CompressionMethodOptions (and CompressionMethodRoutine)
> 
> or
> 
>CompressionOptions (and CompressionRoutine)

Sounds good, thanks.

> 
> OK. But then I don't understand why tsvector.c does things like
> 
> VARSIZE(data) - VARHDRSZ_CUSTOM_COMPRESSED - arrsize
> VARRAWSIZE_4B_C(data) - arrsize
> 
> instead of
> 
> VARSIZE_ANY_EXHDR(data) - arrsize
> VARSIZE_ANY(data) - arrsize
> 
> Seems somewhat confusing.
> 

VARRAWSIZE_4B_C returns original size of data, before compression (from
va_rawsize in current postgres, and from va_info in my patch), not size
of the already compressed data, so you can't use VARSIZE_ANY here.

VARSIZE_ANY_EXHDR in current postgres returns VARSIZE-VARHDRSZ, despite
the varlena is compressed or not, so I just kept this behavior for
custom compressed varlenas too. If you look into tuptoaster.c you will
also see lines like 'VARSIZE(attr) - TOAST_COMPRESS_HDRSZ'. So I think
if VARSIZE_ANY_EXHDR will subtract different header sizes then it
should subtract them for usual compressed varlenas too.

> >   
> 
> Hmmm, it still doesn't work for me. See this:
> 
> test=# create extension pg_lz4 ;
> CREATE EXTENSION
> test=# create table t_lz4 (v text compressed lz4);
> CREATE TABLE
> test=# create table t_pglz (v text);
> CREATE TABLE
> test=# insert into t_lz4 select repeat(md5(1::text),300);
> INSERT 0 1
> test=# insert into t_pglz select * from t_lz4;
> INSERT 0 1
> test=# drop extension pg_lz4 cascade;
> NOTICE:  drop cascades to 2 other objects
> DETAIL:  drop cascades to compression options for lz4
> drop cascades to table t_lz4 column v
> DROP EXTENSION
> test=# \c test
> You are now connected to database "test" as user "user".
> test=# insert into t_lz4 select repeat(md5(1::text),300);^C
> test=# select * from t_pglz ;
> ERROR:  cache lookup failed for compression options 16419
> 
> That suggests no recompression happened.

I will check that. Is your extension published somewhere?