pressor_new() and add a fix in
bbstreamer_lz4_compressor_content() and bbstreamer_lz4_compressor_finalize()
to enlarge the buffer if it falls short of the compress bound.
Patch attached.
Thanks,
Dipesh
From ee9b5e4739bf78b39dc34c9fcc76fc731b09b788 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Thu, 31 Mar
Hi,
The changes look good to me.
Thanks,
Dipesh
compression then setting the value of parameter
ZSTD_c_nbWorkers to a value other than 0 will be no-op and
returns an error.
Thanks,
Dipesh
From 688ad1e3f9b43bf911e8c3837497a874e4a6937f Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Mon, 14 Mar 2022 18:39:02 +0530
Subject: [PATCH] support parallel zstd
Hi,
> > It will be good if we can also fix
> > CreateWalTarMethod to support LZ4 and ZSTD.
> Ok we will see, either Dipesh or I will take care of it.
I took a look at the CreateWalTarMethod to support LZ4 compression
for WAL files. The current implementation involves a 3 step to backup
a WAL
> Sure, please find the rebased patch attached.
Thanks, I have validated v2 patch on top of rebased patch.
Thanks,
Dipesh
nks,
Dipesh
From 47a0ef4348747ffa61eccd7954e00f3cf5fc7222 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Thu, 3 Feb 2022 18:31:03 +0530
Subject: [PATCH] support client side compression and decompression using LZ4
---
src/bin/pg_basebackup/Makefile| 1 +
src/bin/pg_basebac
ed on Jeevan Ladhe's v12 patch
for lz4 compression.
Thanks,
Dipesh
From 67e47579e119897c66e6f5f7a5e5e9542399072f Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Thu, 3 Feb 2022 18:31:03 +0530
Subject: [PATCH] support client side compression and decompression using LZ4
---
src/bi
Hi,
> I made a pass over these patches today and made a bunch of minor
> corrections. New version attached. The two biggest things I changed
> are (1) s/gzip_extractor/gzip_compressor/, because I feel like you
> extract an archive like a tarfile, but that is not what is happening
> here, this is
rom: Dipesh Pandit
Date: Mon, 24 Jan 2022 15:28:48 +0530
Subject: [PATCH 1/2] Support for extracting gzip compressed archive
pg_basebackup can support server side compression using gzip. In
order to support plain format backup with option '-Fp' we need to
add support for decompressing the compressed blo
cd9ba19ea64a50012163a2 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Mon, 24 Jan 2022 15:28:48 +0530
Subject: [PATCH 1/2] Support for extracting gzip compressed archive
pg_basebackup can support server side compression using gzip. In
order to support plain format backup with option '-Fp' we need to
documentation changes, too.
yes, I am working on it.
Note: Before applying the patches, please apply Robert's v12 version
of the patches 0001, 0002 and 0003.
Thanks,
Dipesh
From 826a1cbb639afb7e10a20955d3ec64b1bab1fa80 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Thu, 20 Jan 2022 16:38:36 +0530
Sub
testing and
working on updating the test coverage.
Note: Before applying the patch, please apply Robert's v11 version
of the patches 0001 and 0002.
Thanks,
Dipesh
From 737badce26ed05b5cdb64d9ffd1735fef9acbbf8 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 19 Jan 2022 17:11:45 +0530
Hi,
> 1. I've removed several calls to PgArchForceDirScan() in favor of
> calling it at the top of pgarch_ArchiverCopyLoop(). I believe
> there is some disagreement about this change, but I don't think
> we gain enough to justify the complexity. The main reason we
> exit
nd "Set" functions into
an atomic test-and-check-and-set function to avoid a race condition.
I have incorporated these changes and updated a new patch. PFA patch.
Thanks,
Dipesh
From f8983c7b80ff0f8cbc415d4d9a3ad4992925e775 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 8 Sep 202
g a directory scan at the checkpoint and it will make sure
that any missing file gets archived within the checkpoint boundaries.
Please find the attached patch.
Thanks,
Dipesh
From f05b223b368e40594f0ed8440c0704fb7b970ee0 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 8 Sep 2021 21:47:16 +
gresql.org/message-id/AC78607B-9DA6-41F4-B253-840D3DD964BF%40amazon.com
From 7bb0794f3a41dacfada4863c26189ec01e3bee63 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 8 Sep 2021 21:47:16 +0530
Subject: [PATCH] keep trying the next file approach
---
src/backend/access/transam/xlog.c| 20 +++
src/backend/access/tran
. PFA patch.
Thanks,
Dipesh
From e0ce2b85f9122963d820d005ca093e6c667ca7e6 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 8 Sep 2021 21:47:16 +0530
Subject: [PATCH] keep trying the next file approach
---
src/backend/access/transam/xlog.c| 20
src/backend/access/tr
Hi,
Thanks for the feedback.
> Which approach do you think we should use? I think we have decent
> patches for both approaches at this point, so perhaps we should see if
> we can get some additional feedback from the community on which one we
> should pursue further.
In my opinion both the
_segno is equal to this_segno
unless I am missing something.
Thanks,
Dipesh
From 4fb41105ba4ed9d02a2a551bb4f6cf693ec5c31e Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Thu, 2 Sep 2021 17:16:20 +0530
Subject: [PATCH] keep trying the next file approach
---
src/backend/access/transam/xlog.c
se find attached patch v11.
Thanks,
Dipesh
From 9392fd1b82ade933e8127845013bb2940239af68 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Tue, 24 Aug 2021 12:17:34 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status directory to identify the next WAL
file that needs to be archived. Thi
e that we can use to reset the archiver's stored location. This
> > ensures that we'll keep doing directory scans as long as there are
> > timeline/backup history files to process.
>
> Right.
Done.
Please find the attached patch v10.
Thanks,
Dipesh
From c0a4bf3937934e576db49
n't think of any other scenario which
may
result into a race condition unless I am missing something.
I have incorporated the suggestions and updated a new patch. PFA patch v9.
Thanks,
Dipesh
From 04312fa668a56d9860547a02260d68c339747fa8 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed,
directory scan.
PFA patch v8.
Thanks,
Dipesh
From 1248da2cd96394b632e9853d8ff1f23f123a7813 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status directory to identify the next WAL file
atomic flag here is an efficient choice unless
I am
missing something.
Please find the attached patch v7.
Thanks,
Dipesh
From 55c42f851176a75881a55b1c75d624248169b876 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL
Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status directory to identify the next WAL file
that needs to be archived. This directory scan can be minimised by
maintaining the log
> I'm not sure. I think we need the value to be accurate during
> recovery, so I'm not sure whether replayEndTLI would get us there.
> Another approach might be to set ThisTimeLineID on standbys also.
> Actually just taking a fast look at the code I'm not quite sure why
> that isn't happening
Hi,
> I don't really understand why you are storing something in shared
> memory specifically for the archiver. Can't we use XLogCtl's
> ThisTimeLineID instead of storing another copy of the information?
Yes, we can avoid storing another copy of information. We can
use XLogCtl's ThisTimeLineID
We can update the comments here to list the scenarios
where we may need to perform a full directory scan.
I have incorporated these changes and updated a new patch. Please find
the attached patch v5.
Thanks,
Dipesh
From 3e0f690d1b8fd1d07fa503a807ee97cb9ed7377b Mon Sep 17 00:00:00 2001
From: Dipesh
Hi,
> I don't think it's great that we're using up SIGINT for this purpose.
> There aren't that many signals available at the O/S level that we can
> use for our purposes, and we generally try to multiplex them at the
> application layer, e.g. by setting a latch or a flag in shared memory,
>
sign for another
> thread.
>
> Nathan
>
>
From c597411b08377ea64634d5198071060ec2b9c524 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status directory to identif
e scope to local for "dirScan" and "nextLogSegNo".
PFA patch v3.
Thanks,
Dipesh
From 76260a2ebf90fd063e06dac701e560a506b7a2b7 Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WA
these changes and updated a new patch. PFA, patch v2.
Thanks,
Dipesh
From 22b0e8fb9c778fbfdc945a647f82f6bbd8d6ec0a Mon Sep 17 00:00:00 2001
From: Dipesh Pandit
Date: Wed, 30 Jun 2021 14:05:58 +0530
Subject: [PATCH] mitigate directory scan for WAL archiver
WAL archiver scans the status dire
> specifically about history files being given higher priority for
> archiving. If we go with this change then we'd at least want to rewrite
> or remove those comments, but I don't actually agree that we should
> remove that preference to archive history files ahead of WAL, for the
> reasons
ation sof file system to return just the
> > needed files with ORDER BY ... LIMIT as we already know how to make
> > lookups in database fast ?
>
> Archiving needs to work on a standby so that doesn't seem like an
> option.
>
> Regards,
>
> Andres Freund
>
>
>
Hi,
I have repeated the experiment with 8K block size and found that the
results are not varying much after applying the patch.
Please find the details below.
*Backup type*: local backup using pg_basebackup
*Data size*: Around 200GB (200 tables - each table around 1.05 GB)
*TAR_SEND_SIZE value*:
Hi Asif,
I am reviewing your recent patch and found the patch is not applicable on
latest master.
Could you please resolve the conflicts and update a new patch?
Thanks,
Dipesh
EnterpriseDB: http://www.enterprisedb.com
36 matches
Mail list logo
