Re: Zstandard support for toast compression

2023-05-23 Thread Michael Paquier
On Tue, May 23, 2023 at 05:56:13PM +0300, Aleksander Alekseev wrote: > TWIMC there is an ongoing discussion [1] of making TOAST pointers > extendable since this is a dependency for several patches that are > currently in development. Thanks for the ping. I have seen and read the other thread,

Re: Zstandard support for toast compression

2023-05-23 Thread Aleksander Alekseev
Hi, > Yeah - I think we had better reserve the fourth bit pattern for > something extensible e.g. another byte or several to specify the > actual method, so that we don't have a hard limit of 4 methods. TWIMC there is an ongoing discussion [1] of making TOAST pointers extendable since this is a

Re: Zstandard support for toast compression

2022-05-24 Thread Nikolay Shaplov
В письме от пятница, 20 мая 2022 г. 23:17:42 MSK пользователь Stephen Frost написал: > While I generally agree that we want to provide extensibility in this > area, given that we already have zstd as a compile-time option and it > exists in other parts of the system, I don't think it makes sense

Re: Zstandard support for toast compression

2022-05-23 Thread Robert Haas
On Fri, May 20, 2022 at 4:17 PM Stephen Frost wrote: > A thought I've had before is that it'd be nice to specify a particular > compression method on a data type basis. Wasn't the direction that this > was taken, for reasons, but I wonder about perhaps still having a data > type compression

Re: Zstandard support for toast compression

2022-05-23 Thread Robert Haas
On Mon, May 23, 2022 at 12:33 AM Michael Paquier wrote: > On Thu, May 19, 2022 at 04:12:01PM -0400, Robert Haas wrote: > > On Thu, May 19, 2022 at 4:20 AM Michael Paquier wrote: > >> Btw, shouldn't we have something a bit more, err, extensible for the > >> design of an extensible varlena header?

Re: Zstandard support for toast compression

2022-05-22 Thread Michael Paquier
On Thu, May 19, 2022 at 04:12:01PM -0400, Robert Haas wrote: > On Thu, May 19, 2022 at 4:20 AM Michael Paquier wrote: >> Btw, shouldn't we have something a bit more, err, extensible for the >> design of an extensible varlena header? If we keep it down to some >> bitwise information, we'd be fine

Re: Zstandard support for toast compression

2022-05-20 Thread Stephen Frost
Greetings, * Nikolay Shaplov (dh...@nataraj.su) wrote: > В письме от вторник, 17 мая 2022 г. 23:01:07 MSK пользователь Tom Lane > написал: > > Hi! I came to this branch looking for a patch to review, but I guess I would > join the discussion instead of reading the code. Seems that's what

Re: Zstandard support for toast compression

2022-05-19 Thread Robert Haas
On Thu, May 19, 2022 at 4:20 AM Michael Paquier wrote: > Btw, shouldn't we have something a bit more, err, extensible for the > design of an extensible varlena header? If we keep it down to some > bitwise information, we'd be fine for a long time but it would be > annoying to review again an

Re: Zstandard support for toast compression

2022-05-19 Thread Nikolay Shaplov
В письме от вторник, 17 мая 2022 г. 23:01:07 MSK пользователь Tom Lane написал: Hi! I came to this branch looking for a patch to review, but I guess I would join the discussion instead of reading the code. > >> Yeah - I think we had better reserve the fourth bit pattern for > >> something

Re: Zstandard support for toast compression

2022-05-19 Thread Amit Kapila
On Wed, May 18, 2022 at 9:47 PM Robert Haas wrote: > > I do understand that Zstandard is a good compression algorithm, and if > we had an extensibility mechanism here where one of the four possible > bit patterns then indicates that the next byte (or two or four) stores > the real algorithm type,

Re: Zstandard support for toast compression

2022-05-19 Thread Michael Paquier
On Wed, May 18, 2022 at 12:17:16PM -0400, Robert Haas wrote: > i.e. something like: > > 00 = PGLZ > 01 = LZ4 > 10 = reserved for future emergencies > 11 = extended header with additional type byte (1 of 256 possible > values reserved for Zstandard) Btw, shouldn't we have something a bit more,

Re: Zstandard support for toast compression

2022-05-18 Thread Peter Geoghegan
On Wed, May 18, 2022 at 9:17 AM Robert Haas wrote: > But I want to point out here that you haven't really offered any kind > of argument in favor of supporting Zstd. You basically seem to just be > arguing that it's dumb to worry about running out of bit space, and I > think that's just obviously

Re: Zstandard support for toast compression

2022-05-18 Thread Justin Pryzby
On Tue, May 17, 2022 at 02:54:28PM -0400, Robert Haas wrote: > I don't particularly have anything against adding Zstandard > compression here, but I wonder whether there's any rush. If we decide > not to add this now, we can always change our minds and add it later, > but if we decide to add it

Re: Zstandard support for toast compression

2022-05-18 Thread Robert Haas
On Tue, May 17, 2022 at 4:12 PM Stephen Frost wrote: > I'm getting a bit of deja-vu here from when I was first trying to add > TRUNCATE as a GRANT'able option and being told we didn't want to burn > those precious bits. Right, it's the same issue ... although in that case there are a lot more

Re: Zstandard support for toast compression

2022-05-17 Thread Michael Paquier
On Tue, May 17, 2022 at 04:12:14PM -0400, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> I'm with Robert in that I do not see any urgency to add another method. Okay. >> The fact that Stephen is already questioning whether LZ4 should have >> been added first is not making me

Re: Zstandard support for toast compression

2022-05-17 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Robert Haas (robertmh...@gmail.com) wrote: > >> Yeah - I think we had better reserve the fourth bit pattern for > >> something extensible e.g. another byte or several to specify the > >> actual method, so that we

Re: Zstandard support for toast compression

2022-05-17 Thread Tom Lane
Stephen Frost writes: > * Robert Haas (robertmh...@gmail.com) wrote: >> Yeah - I think we had better reserve the fourth bit pattern for >> something extensible e.g. another byte or several to specify the >> actual method, so that we don't have a hard limit of 4 methods. But >> even with such a

Re: Zstandard support for toast compression

2022-05-17 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, May 17, 2022 at 12:19 AM Michael Paquier wrote: > > Toast compression is supported for LZ4, and thanks to the refactoring > > work done with compression methods assigned to an attribute, adding > > support for more methods is

Re: Zstandard support for toast compression

2022-05-17 Thread Robert Haas
On Tue, May 17, 2022 at 12:19 AM Michael Paquier wrote: > Toast compression is supported for LZ4, and thanks to the refactoring > work done with compression methods assigned to an attribute, adding > support for more methods is straight-forward, as long as we don't > support more than 4 methods

Zstandard support for toast compression

2022-05-16 Thread Michael Paquier
Hi all, Toast compression is supported for LZ4, and thanks to the refactoring work done with compression methods assigned to an attribute, adding support for more methods is straight-forward, as long as we don't support more than 4 methods as the compression ID is stored within the first 2 bits