Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Jim Nasby
On 12/16/15 7:03 AM, Tomas Vondra wrote: While versioning or increasing the 1GB limit are interesting, they have nothing to do with this particular patch. (BTW I don't see how the versioning would work at varlena level - that's something that needs to be handled at data type level). Right, but

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Michael Paquier
On Wed, Dec 16, 2015 at 10:17 PM, Andres Freund wrote: > Again. Unless I miss something that was solved in > www.postgresql.org/message-id/flat/20130615102028.gk19...@alap2.anarazel.de > > Personally I think we should add lz4 and maybe on strongly compressing > algorithm and be done with it. Runti

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Andres Freund
On 2015-12-16 14:14:36 +0100, Tomas Vondra wrote: > I think the main obstacle to make this possible is the lack of free space in > varlena header / need to add the ID of the compression method into the > value. > > FWIW I'd like to allow this (mixing compression types), but I don't think > it's wo

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
Hi, On 12/13/2015 06:28 PM, Alexander Korotkov wrote: > Compression method of column would be stored in pg_attribute table. Dependencies between columns and compression methods would be tracked in pg_depend preventing dropping compression method which is currently in use. Compression method of t

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
On 12/14/2015 07:28 PM, Jim Nasby wrote: On 12/14/15 12:50 AM, Craig Ringer wrote: The issue with per-Datum is that TOAST claims two bits of a varlena header, which already limits us to 1 GiB varlena values, something people are starting to find to be a problem. There's no wiggle room to steal

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
Hi, On 12/14/2015 12:51 PM, Simon Riggs wrote: On 13 December 2015 at 17:28, Alexander Korotkov mailto:a.korot...@postgrespro.ru>> wrote: it would be nice to make compression methods pluggable. Agreed. My thinking is that this should be combined with work to make use of the compressed da

Re: [HACKERS] Proposal: custom compression methods

2015-12-14 Thread Andres Freund
On 2015-12-14 14:50:57 +0800, Craig Ringer wrote: > http://www.postgresql.org/message-id/flat/20130615102028.gk19...@alap2.anarazel.de#20130615102028.gk19...@alap2.anarazel.de > The issue with per-Datum is that TOAST claims two bits of a varlena header, > which already limits us to 1 GiB varlena v

Re: [HACKERS] Proposal: custom compression methods

2015-12-14 Thread Jim Nasby
On 12/14/15 12:50 AM, Craig Ringer wrote: The issue with per-Datum is that TOAST claims two bits of a varlena header, which already limits us to 1 GiB varlena values, something people are starting to find to be a problem. There's no wiggle room to steal more bits. If you want pluggable compressio

Re: [HACKERS] Proposal: custom compression methods

2015-12-14 Thread Simon Riggs
On 13 December 2015 at 17:28, Alexander Korotkov wrote: > it would be nice to make compression methods pluggable. > Agreed. My thinking is that this should be combined with work to make use of the compressed data, which is why Alvaro, Tomas, David have been working on Col Store API for about 1

Re: [HACKERS] Proposal: custom compression methods

2015-12-14 Thread Bill Moran
On Mon, 14 Dec 2015 14:50:57 +0800 Craig Ringer wrote: > On 14 December 2015 at 01:28, Alexander Korotkov > wrote: > > > Hackers, > > > > I'd like to propose a new feature: "Custom compression methods". I missed the initial post on this thread ... Have you started or do you plan to actually s

Re: [HACKERS] Proposal: custom compression methods

2015-12-13 Thread Craig Ringer
On 14 December 2015 at 15:27, Chapman Flack wrote: > On 12/14/15 01:50, Craig Ringer wrote: > > > pg_upgrade means you can't just redefine the current toast bits so the > > compressed bit means "data is compressed, check first byte of varlena > data > > for algorithm" because existing data won't

Re: [HACKERS] Proposal: custom compression methods

2015-12-13 Thread Chapman Flack
On 12/14/15 01:50, Craig Ringer wrote: > pg_upgrade means you can't just redefine the current toast bits so the > compressed bit means "data is compressed, check first byte of varlena data > for algorithm" because existing data won't have that, the first byte will > be the start of the compressed

Re: [HACKERS] Proposal: custom compression methods

2015-12-13 Thread Craig Ringer
On 14 December 2015 at 01:28, Alexander Korotkov wrote: > Hackers, > > I'd like to propose a new feature: "Custom compression methods". > Are you aware of the past work in this area? There's quite a bit of history and I strongly advise you to read the relevant threads to make sure you don't run

[HACKERS] Proposal: custom compression methods

2015-12-13 Thread Alexander Korotkov
Hackers, I'd like to propose a new feature: "Custom compression methods". *Motivation* Currently when datum doesn't fit the page PostgreSQL tries to compress it using PGLZ algorithm. Compression of particular attributes could be turned on/off by tuning storage parameter of column. Also, there is