Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-06 Thread LasseReinhold
Stephen R. van den Berg wrote: I asked the author of the QuickLZ algorithm about licensing... Sounds like he is willing to cooperate. This is what I got from him: On Sat, Jan 3, 2009 at 17:56, Lasse Reinhold l...@quicklz.com wrote: Hi Stephen, That sounds really exciting, I'd love to

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-06 Thread Robert Haas
not compressing very small datums ( 256 bytes) also seems smart, since that could end up producing a lot of extra compression attempts, most of which will end up saving little or no space. That was presumably the rationale for the original logic. However experience shows that there are

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-06 Thread Alvaro Herrera
Robert Haas escribió: char(n) is kind of a wierd case because you could also compress by storing a count of the trailing spaces, without applying a general-purpose compression algorithm. But either way the field is no longer fixed-width, and therefore field access can't be done as a simple

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Douglas McNaught
On Mon, Jan 5, 2009 at 3:18 AM, Stephen R. van den Berg s...@cuci.nl wrote: I'm not speaking for Lasse, merely providing food for thought, but it sounds feasible to me (and conforming to Lasse's spirit of his intended license) to put something like the following license on his code, which would

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Andrew Dunstan
Douglas McNaught wrote: On Mon, Jan 5, 2009 at 3:18 AM, Stephen R. van den Berg s...@cuci.nl wrote: I'm not speaking for Lasse, merely providing food for thought, but it sounds feasible to me (and conforming to Lasse's spirit of his intended license) to put something like the following

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Stefan Kaltenbrunner
Andrew Dunstan wrote: Douglas McNaught wrote: On Mon, Jan 5, 2009 at 3:18 AM, Stephen R. van den Berg s...@cuci.nl wrote: I'm not speaking for Lasse, merely providing food for thought, but it sounds feasible to me (and conforming to Lasse's spirit of his intended license) to put something

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Robert Haas
Are we willing to be dropped from Debian and possibly Red Hat if this is the case? No. I frankly think this discussion is a dead end. The whole thing got started because Alex Hunsaker pointed out that his database got a lot bigger because we disabled compression on columns 1MB. It seems

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Stephen R. van den Berg
Douglas McNaught wrote: Grant license to use the code in question without cost, provided that the code is being linked to at least 50% of the PostgreSQL code it is being distributed alongside with. This should allow commercial reuse in derived products without undesirable sideeffects. I

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Tom Lane
Douglas McNaught d...@mcnaught.org writes: I think Postgres becomes non-DFSG-free if this is done. All of a sudden one can't pull arbitrary pieces of code out of PG and use them in other projects (which I'd argue is the intent if not the letter of the DFSG). Have we ever allowed code in on

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Andrew Chernow
Robert Haas wrote: What we do have is a suggestion from several people that the database shouldn't be in the business of compressing data AT ALL. If we want +1 IMHO, this is a job for the application. I also think the current implementation is a little odd in that it only compresses data

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread A.M.
On Jan 5, 2009, at 1:16 PM, Stephen R. van den Berg wrote: Upon reading the DFSG, it seems you have a point... However... QuickLZ is dual licensed: a. Royalty-free-perpetuous-use as part of the PostgreSQL backend or any derived works of PostgreSQL which link in *at least* 50% of the

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Gregory Stark
Robert Haas robertmh...@gmail.com writes: Regardless of whether we do that or not, no one has offered any justification of the arbitrary decision not to compress columns 1MB, Er, yes, there was discussion before the change, for instance:

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: The whole thing got started because Alex Hunsaker pointed out that his database got a lot bigger because we disabled compression on columns 1MB. It seems like the obvious thing to do is turn it back on again. I suggest that before we make any

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Robert Haas
On Mon, Jan 5, 2009 at 2:02 PM, Gregory Stark st...@enterprisedb.com wrote: Regardless of whether we do that or not, no one has offered any justification of the arbitrary decision not to compress columns 1MB, Er, yes, there was discussion before the change, for instance:

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Holger Hoffstaette
On Mon, 05 Jan 2009 13:44:57 -0500, Andrew Chernow wrote: Robert Haas wrote: What we do have is a suggestion from several people that the database shouldn't be in the business of compressing data AT ALL. If we want DB2 users generally seem very happy with the built-in compression. IMHO,

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Stephen R. van den Berg
Alvaro Herrera wrote: On Sat, Jan 3, 2009 at 17:56, Lasse Reinhold l...@quicklz.com wrote: That sounds really exciting, I'd love to see QuickLZ included into PostgreSQL. I'd be glad to offer support and add custom optimizations, features or hacks or whatever should turn up. My only

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Joshua D. Drake
On Mon, 2009-01-05 at 13:04 -0500, Robert Haas wrote: Are we willing to be dropped from Debian and possibly Red Hat if this is the case? Regardless of whether we do that or not, no one has offered any justification of the arbitrary decision not to compress columns 1MB, and at least one

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Mark Mielke
Guaranteed compression of large data fields is the responsibility of the client. The database should feel free to compress behind the scenes if it turns out to be desirable, but an expectation that it compresses is wrong in my opinion. That said, I'm wondering why compression has to be a

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Stephen R. van den Berg
A.M. wrote: On Jan 5, 2009, at 1:16 PM, Stephen R. van den Berg wrote: Upon reading the DFSG, it seems you have a point... However... QuickLZ is dual licensed: a. Royalty-free-perpetuous-use as part of the PostgreSQL backend or any derived works of PostgreSQL which link in *at least* 50% of the

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Gregory Stark
Mark Mielke m...@mark.mielke.cc writes: It seems to me that transparent file system compression doesn't have limits like files must be less than 1 Mbyte to be compressed. They don't exhibit poor file system performance. Well I imagine those implementations are more complex than toast is. I'm

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Stephen R. van den Berg
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: The whole thing got started because Alex Hunsaker pointed out that his database got a lot bigger because we disabled compression on columns 1MB. It seems like the obvious thing to do is turn it back on again. After poking around in

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Tom Lane
Gregory Stark st...@enterprisedb.com writes: Hm, It occurs to me we could almost use the existing code. Just store it as a regular uncompressed external datum but allow the toaster to operate on the data column (which it's normally not allowed to) to compress it, but not store it externally.

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Mark Mielke
Gregory Stark wrote: Mark Mielke m...@mark.mielke.cc writes: It seems to me that transparent file system compression doesn't have limits like files must be less than 1 Mbyte to be compressed. They don't exhibit poor file system performance. Well I imagine those implementations are

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Andrew Chernow
Holger Hoffstaette wrote: On Mon, 05 Jan 2009 13:44:57 -0500, Andrew Chernow wrote: Robert Haas wrote: What we do have is a suggestion from several people that the database shouldn't be in the business of compressing data AT ALL. If we want DB2 users generally seem very happy with the

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Robert Haas
I suggest that before we make any knee-jerk responses, we need to go back and reread the prior discussion. http://archives.postgresql.org/pgsql-patches/2008-02/msg00053.php and that message links to several older threads that were complaining about the 8.3 behavior. In particular the notion

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Bruce Momjian
Robert Haas wrote: After poking around in those threads a bit, I think that the current threshold of 1MB was something I just made up on the fly (I did note that it needed tuning...). Perhaps something like 10MB would be a better default. Another possibility is to have different minimum

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: After reading these discussions, I guess I still don't understand why we would treat small and large datums differently. It seems to me that you had it about right here: http://archives.postgresql.org/pgsql-hackers/2007-08/msg00082.php # Or maybe it

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-05 Thread Gregory Stark
Robert Haas robertmh...@gmail.com writes: not compressing very small datums ( 256 bytes) also seems smart, since that could end up producing a lot of extra compression attempts, most of which will end up saving little or no space. That was presumably the rationale for the original logic.

[HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-04 Thread Stephen R. van den Berg
I asked the author of the QuickLZ algorithm about licensing... Sounds like he is willing to cooperate. This is what I got from him: On Sat, Jan 3, 2009 at 17:56, Lasse Reinhold l...@quicklz.com wrote: Hi Stephen, That sounds really exciting, I'd love to see QuickLZ included into PostgreSQL.

Re: [HACKERS] QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

2009-01-04 Thread Alvaro Herrera
On Sat, Jan 3, 2009 at 17:56, Lasse Reinhold l...@quicklz.com wrote: That sounds really exciting, I'd love to see QuickLZ included into PostgreSQL. I'd be glad to offer support and add custom optimizations, features or hacks or whatever should turn up. My only concern is to avoid