On Thu, Mar 17, 2022 at 06:12:20AM -0500, Justin Pryzby wrote:
> I think this should use
>
> +#include "lz4frame.h"
>
> commit ba595d2322da095a1e6703171b3f1f2815cb
> Author: Michael Paquier
> Date: Fri Nov 5 11:33:25 2021 +0900
>
> Add support for LZ4 compression in pg_receivewal
On Sat, Feb 12, 2022 at 12:52:40PM +0900, Michael Paquier wrote:
> On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
> > Over in
> > http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
> > I was noticing that CreateWalTarMethod doesn't support LZ4
> >
On Fri, Feb 11, 2022 at 10:52 PM Michael Paquier wrote:
> > And, on a related note, Michael, do you plan to get something
> > committed here?
>
> Apart from f79962d, ba5 and 50e1441, I don't think that there was
> something left to do for this thread. Perhaps I am missing something?
Oh, my
On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
> Over in
> http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
> I was noticing that CreateWalTarMethod doesn't support LZ4
> compression. It would be nice if it did. I thought maybe the patch on
>
On Thu, Nov 4, 2021 at 10:47 PM Michael Paquier wrote:
> Indeed. My rebase was a bit sloppy here.
Hi!
Over in
http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
I was noticing that CreateWalTarMethod doesn't support LZ4
compression. It would be nice if it
On Wed, Nov 24, 2021 at 10:55 AM Michael Paquier
wrote:
> On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
> > On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
> > wrote:
> >> Fair enough. But, still I have a doubt in mind what benefit would that
> >> really bring to us here, because
On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
> On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
> wrote:
>> Fair enough. But, still I have a doubt in mind what benefit would that
>> really bring to us here, because we are immediately also freeing the
>> lz4buf without using it
On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
wrote:
> Fair enough. But, still I have a doubt in mind what benefit would that
> really bring to us here, because we are immediately also freeing the
> lz4buf without using it anywhere.
Yeah, I'm also doubtful about that. If we're freeng the
On Fri, Nov 19, 2021 at 7:37 AM Michael Paquier wrote:
> On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
> > In dir_open_for_write() I observe that we are writing the header
> > and then calling LZ4F_compressEnd() in case there is an error
> > while writing the buffer to the file,
‐‐‐ Original Message ‐‐‐
On Friday, November 19th, 2021 at 3:07 AM, Michael Paquier
wrote:
> On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
>
> > In dir_open_for_write() I observe that we are writing the header
> > and then calling LZ4F_compressEnd() in case there is an
On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
> In dir_open_for_write() I observe that we are writing the header
> and then calling LZ4F_compressEnd() in case there is an error
> while writing the buffer to the file, and the output buffer of
> LZ4F_compressEnd() is not written
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the
‐‐‐ Original Message ‐‐‐
On Friday, November 5th, 2021 at 3:47 AM, Michael Paquier
wrote:
>
> I have spent an extra couple of hours staring at the code, and the
> whole looked fine, so applied. While on it, I have tested the new TAP
> tests with all the possible combinations of
On Thu, Nov 04, 2021 at 05:02:28PM +, gkokola...@pm.me wrote:
> Removed an extra condinional check while switching over compression_method.
Indeed. My rebase was a bit sloppy here.
> because compression_method is the global option exposed to the whereas
> wal_compression_method is the local
ng
emitted
during the parsing of such a file.
Cheers,
//Georgios
> --
> MichaelFrom 48720e7c6ba771c45d43dc9f5e6833f8bb6715e6 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos
Date: Thu, 4 Nov 2021 16:05:21 +
Subject: [PATCH v12] Teach pg_receivewal to use LZ4 compression
The program pg_
‐‐‐ Original Message ‐‐‐
On Thursday, November 4th, 2021 at 9:21 AM, Michael Paquier
wrote:
> On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
> Thanks. I have looked at 0001 today, and applied it after fixing a
> couple of issues.
Great! Thank you very much.
>
s stuff tomorrow, but
this looks rather committable overall.
--
Michael
From d2d76d8fefb1bad5db611746cf3d0ac89a67de4b Mon Sep 17 00:00:00 2001
From: Michael Paquier
Date: Thu, 4 Nov 2021 17:19:41 +0900
Subject: [PATCH v11] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compress
021 16:24:48 +0900
Subject: [PATCH v10] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received
WAL. This commit teaches it to also be able to use LZ4 compression. It
is required that the binary is build using the -llz4 flag. It is enabled
shortly.
Please find v9 attached.
Cheers,
//Georgios
>
> > MichaelFrom 8e33136f81c3197020053cba0f7f070d594f056e Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos
Date: Wed, 3 Nov 2021 08:59:58 +
Subject: [PATCH v9 2/2] Teach pg_receivewal to use LZ4 compression
The program pg_rec
‐‐‐ Original Message ‐‐‐
On Wednesday, November 3rd, 2021 at 12:23 AM, Michael Paquier
wrote:
> On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
>> On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander mag...@hagander.net wrote:
>>> I think for the end user, it is strictly
On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
> On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander wrote:
>> I think for the end user, it is strictly better to name it "gzip",
>> and given that the target of this option is the end user we should
>> do so. (It'd be different it we were
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander wrote:
> Um, why?
>
> That we are using zlib to provide the compression is an implementation
> detail. Whereas AFAIK "gzip" refers to both the program and the format. And
> we specifically use the gzxxx() functions in zlib, in order to produce gzip
On Tue, Nov 2, 2021 at 9:51 AM Michael Paquier wrote:
> On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
> > On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> > > Agreed.
> > >
> > > I have already started on v8 of the patch with that technique. I should
> > >
-95,8 +101,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(cons
On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
> On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> > Agreed.
> >
> > I have already started on v8 of the patch with that technique. I should
> > be able to update the thread soon.
>
> Nice, thanks!
By the way, I
On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> Agreed.
>
> I have already started on v8 of the patch with that technique. I should
> be able to update the thread soon.
Nice, thanks!
--
Michael
signature.asc
Description: PGP signature
‐‐‐ Original Message ‐‐‐
On Monday, November 1st, 2021 at 9:09 AM, Michael Paquier
wrote:
> On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
>
> It would be good to test with many segments, but could we think about
> just relying on LZ4F_decompress() with a frame and
On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
> Why would the header size change between the moment the segment is
> begun and it is finished? We could store it in memory and write it
> again when the segment is closed instead, even if it means to fseek()
> back to the
On Fri, Oct 29, 2021 at 09:45:41AM +, gkokola...@pm.me wrote:
> On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier
> wrote:
>> We don't really care about contentSize as long as a segment is not
>> completed. Rather than filling contentSize all the time we write
>> something, we'd
when closing a completed segment. Could you look at it?
>
Thank you. Find v7 attached, rebased to the current head.
Cheers,
//Georgios
> Thanks,
> --
> MichaeFrom c3c2eca22102cd0186eb1975339248a200e1ceb9 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos
Date: Fri, 22 Oct 2021 13:14:15 +
S
. Could you look at it?
Thanks,
--
Michael
From 37e3800d279566445864ed82f29e8d650c72d8cd Mon Sep 17 00:00:00 2001
From: Michael Paquier
Date: Sat, 18 Sep 2021 15:11:49 +0900
Subject: [PATCH v6] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
‐‐‐ Original Message ‐‐‐
On Friday, September 17th, 2021 at 09:39, Michael Paquier
wrote:
> On Thu, Sep 16, 2021 at 03:17:15PM +, gkokola...@pm.me wrote:
>
> > Hopefully fixed.
>
> Thanks for the new version. I have put my hands on the patch, and
> began reviewing its internals
On Thu, Sep 16, 2021 at 03:17:15PM +, gkokola...@pm.me wrote:
> Hopefully fixed.
Thanks for the new version. I have put my hands on the patch, and
began reviewing its internals with LZ4. I am not done with it yet,
and I have noticed some places that could be improved (error handling,
some
‐‐‐ Original Message ‐‐‐
On Wednesday, September 15th, 2021 at 08:46, Michael Paquier
mich...@paquier.xyz wrote:
Hi,
thank you for the review.
> On Fri, Sep 10, 2021 at 08:21:51AM +, gkokola...@pm.me wrote:
>
> > Agreed. A default value of 5, which is in the middle point of
On Fri, Sep 10, 2021 at 08:21:51AM +, gkokola...@pm.me wrote:
> Agreed. A default value of 5, which is in the middle point of options, has
> been
> defined and used.
>
> In addition, the tests have been adjusted to mimic the newly added gzip tests.
Looking at lz4frame.h, there is
‐‐‐ Original Message ‐‐‐
On Saturday, September 11th, 2021 at 07:02, Jian Guo wrote:
Hi,
thank you for looking at the patch.
> - LZ4F_decompressionContext_t ctx = NULL;
> - snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir,
> dirent->d_name);
> -
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed
to be
* completed. For non-compressed segments just check the
on-disk size
-* and see if it matches a
‐‐‐ Original Message ‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier wrote:
Hi,
please find v3 of the patch attached, rebased to the current head.
> > Michael Paquier wrote:
> >
>
> * http://www.zlib.org/rfc-gzip.html.
>
> - - For lz4 compressed segments
> */
>
On Mon, Jul 12, 2021 at 3:15 PM Magnus Hagander wrote:
>
> On Mon, Jul 12, 2021 at 11:33 AM wrote:
> >
> >
> >
> > ‐‐‐ Original Message ‐‐‐
> >
> > On Monday, July 12th, 2021 at 07:56, Michael Paquier
> > wrote:
> >
> > > On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> >
On Mon, Jul 12, 2021 at 11:33 AM wrote:
>
>
>
> ‐‐‐ Original Message ‐‐‐
>
> On Monday, July 12th, 2021 at 07:56, Michael Paquier
> wrote:
>
> > On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> >
> > > On Thu, Jul 8, 2021 at 7:48 PM gkokola...@pm.me wrote:
> > >
> > > > We
‐‐‐ Original Message ‐‐‐
On Monday, July 12th, 2021 at 07:56, Michael Paquier
wrote:
> On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
>
> > On Thu, Jul 8, 2021 at 7:48 PM gkokola...@pm.me wrote:
> >
> > > We can, though I am not in favour of doing so. There is
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> On Thu, Jul 8, 2021 at 7:48 PM wrote:
>> We can, though I am not in favour of doing so. There is seemingly
>> little benefit for added complexity.
>
> I am really not sure what complexity you are talking about, do you
> mean since
On Thu, Jul 8, 2021 at 7:48 PM wrote:
>
> Dilip Kumar wrote:
>
> > Wouldn't it be better to call it compression method instead of
> > compression program?
>
> Agreed. This is inline with the suggestions of other reviewers.
> Find the change in the attached patch.
Thanks, I will have a look.
> >
‐‐‐ Original Message ‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier wrote:
> On Thu, Jul 08, 2021 at 02:18:40PM +, gkokola...@pm.me wrote:
>
> > please find v2 of the patch which tries to address the commends
> >
> > received so far.
>
> Thanks!
>
> > Michael Paquier
On Thu, Jul 08, 2021 at 02:18:40PM +, gkokola...@pm.me wrote:
> please find v2 of the patch which tries to address the commends
> received so far.
Thanks!
> Michael Paquier wrote:
>> + system_or_bail('lz4', '-t', $lz4_wals[0]);
>> I think that you should just drop this part of the test.
Hi,
please find v2 of the patch which tries to address the commends received so far.
Thank you all for your comments.
Michael Paquier wrote:
> Documentation is missing from the patch.
>
It has now been added.
> + LZ4F_compressionContext_t ctx;
> + size_t outbufCapacity;
> + void
‐‐‐ Original Message ‐‐‐
On Friday, July 2nd, 2021 at 03:10, Michael Paquier wrote:
> On Thu, Jul 01, 2021 at 02:10:17PM +, gkokola...@pm.me wrote:
>
> > Micheal suggested on the same thread to move my entry in the help output so
> > that
> >
> > the output remains ordered. I
On Thu, Jul 01, 2021 at 02:10:17PM +, gkokola...@pm.me wrote:
> Micheal suggested on the same thread to move my entry in the help output so
> that
> the output remains ordered. I would like the options for the compression
> method and
> the already existing compression level to next to each
‐‐‐ Original Message ‐‐‐
On Thursday, July 1st, 2021 at 15:58, Magnus Hagander
wrote:
> On Thu, Jul 1, 2021 at 3:39 PM gkokola...@pm.me wrote:
>
> > ‐‐‐ Original Message ‐‐‐
> >
> > On Thursday, July 1st, 2021 at 12:28, Magnus Hagander mag...@hagander.net
> > wrote:
> >
> >
On Thu, Jul 1, 2021 at 3:39 PM wrote:
>
>
>
> ‐‐‐ Original Message ‐‐‐
>
> On Thursday, July 1st, 2021 at 12:28, Magnus Hagander
> wrote:
>
> > On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbal...@gmail.com wrote:
> >
> > > On Tue, Jun 29, 2021 at 8:15 PM gkokola...@pm.me wrote:
> >
‐‐‐ Original Message ‐‐‐
On Thursday, July 1st, 2021 at 12:28, Magnus Hagander
wrote:
> On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbal...@gmail.com wrote:
>
> > On Tue, Jun 29, 2021 at 8:15 PM gkokola...@pm.me wrote:
> >
> > > Hi,
> > >
> > > The program pg_receivewal can use
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar wrote:
>
> On Tue, Jun 29, 2021 at 8:15 PM wrote:
> >
> > Hi,
> >
> > The program pg_receivewal can use gzip compression to store the received
> > WAL.
> > This patch teaches it to be able to use lz4 compression if the binary is
> > build
> > using
On Tue, Jun 29, 2021 at 8:15 PM wrote:
>
> Hi,
>
> The program pg_receivewal can use gzip compression to store the received WAL.
> This patch teaches it to be able to use lz4 compression if the binary is build
> using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
1.
@@
On Tue, Jun 29, 2021 at 02:45:17PM +, gkokola...@pm.me wrote:
> The program pg_receivewal can use gzip compression to store the received WAL.
> This patch teaches it to be able to use lz4 compression if the binary is build
> using the -llz4 flag.
Nice.
> Previously, the user had to use the
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression
55 matches
Mail list logo
