Re: [HACKERS] pluggable compression support

2013-10-02 Thread Robert Haas
On Tue, Oct 1, 2013 at 9:56 PM, Daniel Farina wrote: > On Mon, Sep 30, 2013 at 1:49 PM, Huchev wrote: >> How come any compressor which could put some competition to pglz is >> systematically pushed out of the field on the ground of unverifiable "legal >> risks" ? > > Because pglz has been around

Re: [HACKERS] pluggable compression support

2013-10-01 Thread Daniel Farina
On Mon, Sep 30, 2013 at 1:49 PM, Huchev wrote: > How come any compressor which could put some competition to pglz is > systematically pushed out of the field on the ground of unverifiable "legal > risks" ? Because pglz has been around for a while and has not caused patent trouble. The risks have

Re: [HACKERS] pluggable compression support

2013-09-30 Thread Huchev
I've been following this issue these last few months. Having the latest and best compressors built-in is a fashionable features these days. And for good reasons. I'm quite amazed that this issue is still considered a "legal risk". To put this in perspective, the *whole world* is using LZ4 by now.

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Josh Berkus
On 06/25/2013 12:23 PM, Tom Lane wrote: > Josh Berkus writes: >> However, can you tell me what exactly you are concerned about? lz4 is >> under the BSD license, and released by Google. Why are we worried, exactly? > > Patents. The license on the code doesn't matter --- worst case, if > someone

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Tom Lane
Josh Berkus writes: > However, can you tell me what exactly you are concerned about? lz4 is > under the BSD license, and released by Google. Why are we worried, exactly? Patents. The license on the code doesn't matter --- worst case, if someone objected, we could rewrite the algorithm ourselves

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Claudio Freire
On Tue, Jun 25, 2013 at 4:15 PM, Andres Freund wrote: >> However, can you tell me what exactly you are concerned about? lz4 is >> under the BSD license, and released by Google. > > Snappy is released/copyrighted by google. lz4 by Yann Collet. > > Both are under BSD licenses (3 and 2 clause variant

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Andres Freund
On 2013-06-25 12:08:22 -0700, Josh Berkus wrote: > On 06/25/2013 11:42 AM, Andres Freund wrote: > > True. So, how do we proceed on that? > > > > The ASF decided it was safe to use lz4 in cassandra. Does anybody have > > contacts over there? > > > > Btw, I have the feeling we hold this topic to a

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Josh Berkus
On 06/25/2013 11:42 AM, Andres Freund wrote: > True. So, how do we proceed on that? > > The ASF decided it was safe to use lz4 in cassandra. Does anybody have > contacts over there? > > Btw, I have the feeling we hold this topic to a higher standard wrt > patent issues than other work in postgres

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Andres Freund
On 2013-06-25 12:22:31 -0400, Robert Haas wrote: > On Thu, Jun 20, 2013 at 8:09 PM, Andres Freund wrote: > > On 2013-06-15 12:20:28 +0200, Andres Freund wrote: > >> On 2013-06-14 21:56:52 -0400, Robert Haas wrote: > >> > I don't think we need it. I think what we need is to decide is which > >> >

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Robert Haas
On Thu, Jun 20, 2013 at 8:09 PM, Andres Freund wrote: > On 2013-06-15 12:20:28 +0200, Andres Freund wrote: >> On 2013-06-14 21:56:52 -0400, Robert Haas wrote: >> > I don't think we need it. I think what we need is to decide is which >> > algorithm is legally OK to use. And then put it in. >> > >

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Alvaro Herrera
Simon Riggs wrote: > On 15 June 2013 12:25, Hannu Krosing wrote: > > > additionally it *could* have the property Simon proposed earlier > > of *uncompressed* pages having some predetermined size, so we > > could retain optimisations of substring() even on compressed TOAST > > values. > > That wa

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Simon Riggs
On 15 June 2013 12:25, Hannu Krosing wrote: > additionally it *could* have the property Simon proposed earlier > of *uncompressed* pages having some predetermined size, so we > could retain optimisations of substring() even on compressed TOAST > values. That wasn't about having fixed size pages,

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Simon Riggs
On 15 June 2013 13:02, Andres Freund wrote: > On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: >> If it were truly pluggable - that is just load a .dll, set a GUG and start >> using it > > Ok. I officially rechristen the patchset to 'extensible compression > support'. +1 (I confess I was confu

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Hannu Krosing
On 06/16/2013 03:50 AM, Robert Haas wrote: > On Sat, Jun 15, 2013 at 8:11 AM, Hannu Krosing wrote: >> Claiming that the algorithm will be one of only two (current and >> "whatever algorithm we come up with ") suggests that it is >> only one bit, which is undoubtedly too little for having a "plugga

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Robert Haas
On Sat, Jun 15, 2013 at 8:22 AM, Hannu Krosing wrote: >> You suddently need to solve the question of how the identifiers for >> compression formats are allocated and preserved across pg_upgrade and >> across machines. > This is something similar we already need to do for any non-builtin type > OID

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Robert Haas
On Sat, Jun 15, 2013 at 8:11 AM, Hannu Krosing wrote: > Claiming that the algorithm will be one of only two (current and > "whatever algorithm we come up with ") suggests that it is > only one bit, which is undoubtedly too little for having a "pluggable" > compression API :) See http://www.postg

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Amit kapila
On Saturday, June 15, 2013 3:50 PM Andres Freund wrote: On 2013-06-14 21:56:52 -0400, Robert Haas wrote: >> I don't think we need it. I think what we need is to decide is which >> algorithm is legally OK to use. And then put it in. >> >> In the past, we've had a great deal of speculation about th

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 14:11:54 +0200, Hannu Krosing wrote: > > Could you perhaps actually read the the description and the discussion > > before making wild suggestions? Possibly even the patch. > > Compressed toast datums now *do* have an identifier of the compression > > algorithm used. > > That's how w

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 02:02 PM, Andres Freund wrote: > On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: >> If it were truly pluggable - that is just load a .dll, set a GUG and start >> using it > Ok. I officially rechristen the patchset to 'extensible compression > support'. > >> - then the selection of

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 01:56 PM, Andres Freund wrote: > On 2013-06-15 13:25:49 +0200, Hannu Krosing wrote: >> On 06/15/2013 02:20 AM, Andres Freund wrote: >>> On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: On 06/14/2013 04:01 PM, Andres Freund wrote: > It still contains a guc as described in the

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: > If it were truly pluggable - that is just load a .dll, set a GUG and start > using it Ok. I officially rechristen the patchset to 'extensible compression support'. > - then the selection of algorithms would be much > wider as several slow-but-h

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 13:25:49 +0200, Hannu Krosing wrote: > On 06/15/2013 02:20 AM, Andres Freund wrote: > > On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: > >> On 06/14/2013 04:01 PM, Andres Freund wrote: > >>> It still contains a guc as described in the above message to control the > >>> algorithm use

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 02:20 AM, Andres Freund wrote: > On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: >> On 06/14/2013 04:01 PM, Andres Freund wrote: >>> It still contains a guc as described in the above message to control the >>> algorithm used for compressing new tuples but I think we should remove >>>

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 03:56 AM, Robert Haas wrote: > On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund wrote: >> On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-14 21:56:52 -0400, Robert Haas wrote: > I don't think we need it. I think what we need is to decide is which > algorithm is legally OK to use. And then put it in. > > In the past, we've had a great deal of speculation about that legal > question from people who are not lawyers. Maybe

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 06:56 PM, Robert Haas wrote: On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always use the

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Robert Haas
On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund wrote: > On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: >> >> > No. I think as long as we only have pglz and one new algorithm (even if >> > that is lz4 instead of the current snappy) we should just always use the >> > new algorithm. Unless I missed

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Andres Freund
On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: > > > No. I think as long as we only have pglz and one new algorithm (even if > > that is lz4 instead of the current snappy) we should just always use the > > new algorithm. Unless I missed it nobody seemed to have voiced a > > contrary position? >

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Josh Berkus
> No. I think as long as we only have pglz and one new algorithm (even if > that is lz4 instead of the current snappy) we should just always use the > new algorithm. Unless I missed it nobody seemed to have voiced a > contrary position? > For testing/evaluation the guc seems to be sufficient. The

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Andres Freund
On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: > On 06/14/2013 04:01 PM, Andres Freund wrote: > > It still contains a guc as described in the above message to control the > > algorithm used for compressing new tuples but I think we should remove > > that guc after testing. > > Did you add the st

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Josh Berkus
On 06/14/2013 04:01 PM, Andres Freund wrote: > It still contains a guc as described in the above message to control the > algorithm used for compressing new tuples but I think we should remove > that guc after testing. Did you add the storage attribute? -- Josh Berkus PostgreSQL Experts Inc. htt