Re: pgBackRest for a 50 TB database

2023-12-19 Thread Abhishek Bhola
e any option to ignore the subscription? Thanks On Thu, Oct 5, 2023 at 10:19 PM Stephen Frost wrote: > Greetings, > > On Thu, Oct 5, 2023 at 03:10 Abhishek Bhola < > abhishek.bh...@japannext.co.jp> wrote: > >> Here is the update with compress-type=zst in the config file >

Change error code severity for syslog?

2023-10-11 Thread Abhishek Bhola
Hi The Postgres errors when sent to syslog have their severity levels translated as follows: https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-SEVERITY-LEVELS I have a system which raises an alert every time syslog receives an ERROR (or higher severity level). Fo

Re: pgBackRest for a 50 TB database

2023-10-05 Thread Abhishek Bhola
perform such a pgBackRest trial from a production DB to > a suitable Storage Device( Mostly Unified storage DELL Unity) > > Any inputs are most welcome. > > Thanks, > Krishane > > On Tue, Oct 3, 2023 at 12:14 PM Abhishek Bhola < > abhishek.bh...@japannext.co.jp> wro

Re: pgBackRest for a 50 TB database

2023-10-03 Thread Abhishek Bhola
Hi Stephen No, I did not try that. Let me try that now and report the numbers here, both in terms of size and time taken. Thanks for the suggestion. On Tue, Oct 3, 2023 at 10:39 PM Stephen Frost wrote: > Greetings, > > On Mon, Oct 2, 2023 at 20:08 Abhishek Bhola < &

Re: pgBackRest for a 50 TB database

2023-10-02 Thread Abhishek Bhola
: 00010001AC0E004E / 00010001AC0E004E database size: 38248.9GB, database backup size: 38248.9GB repo1: backup size: 6222.0GB Hope it helps someone to use these numbers as some reference. Thanks On Mon, Aug 28, 2023 at 12:30 AM Abhishek Bhola < abhishek

Re: pgBackRest for a 50 TB database

2023-08-27 Thread Abhishek Bhola
having published their numbers, that would be great, but if not, I will post them once I set it up. Thanks for your help. Cheers, Abhishek On Mon, Aug 28, 2023 at 12:22 AM Stephen Frost wrote: > Greetings, > > * Abhishek Bhola (abhishek.bh...@japannext.co.jp) wrote: > > I a

pgBackRest for a 50 TB database

2023-08-27 Thread Abhishek Bhola
Hi I am trying to use pgBackRest for all my Postgres servers. I have tested it on a sample database and it works fine. But my concern is for some of the bigger DB clusters, the largest one being 50TB and growing by about 200-300GB a day. I plan to mount NAS storage on my DB server to store my bac

Re: Difference in the tablespace folders on primary and secondary nodes

2023-07-26 Thread Abhishek Bhola
o detect this and skip those files? Please assume for this case that we ran out of capacity to store the WAL segments from the primary while the secondary node was down. Thanks On Wed, 26 Jul 2023, 22:55 Stephen Frost, wrote: > Greetings, > > * Abhishek Bhola (abhishek.bh...@japannex

Re: Difference in the tablespace folders on primary and secondary nodes

2023-07-25 Thread Abhishek Bhola
So for example, if I am able to reindex all the tables, select all the data and vacuum the db, do I still need to pg_dump and pg_restore those DBs? Thanks On Wed, Jul 26, 2023 at 11:00 AM Abhishek Bhola < abhishek.bh...@japannext.co.jp> wrote: > Hi Stephen > > Thank you for your

Re: Difference in the tablespace folders on primary and secondary nodes

2023-07-25 Thread Abhishek Bhola
it to the backup node? Thanks On Tue, Jul 25, 2023 at 9:22 PM Stephen Frost wrote: > Greetings, > > * Abhishek Bhola (abhishek.bh...@japannext.co.jp) wrote: > > I recently set up a Postgres15 master-slave (Primary-secondary) cluster > on > > 2 nodes. At the time of starting, I ensu

Difference in the tablespace folders on primary and secondary nodes

2023-07-25 Thread Abhishek Bhola
I recently set up a Postgres15 master-slave (Primary-secondary) cluster on 2 nodes. At the time of starting, I ensured that data files are exactly the same on both the nodes. The size of the DB is 1.5TB. The directory structure on both nodes looks as follows: ``` - /PROD/datadg/ | |-> /PROD

Re: PG 13.6 : Data corruption error message not being sent to syslog

2022-05-25 Thread Abhishek Bhola
But the error doesn't consist of any of the above codes, so why is it being skipped? On Wed, 25 May 2022, 23:11 Tom Lane, wrote: > Abhishek Bhola writes: > > This morning I noticed this error in my PG CSV log file. > > 2022-05-25 23:59:17.776 JST,,,117110,,628cf2c5.1c976,1

PG 13.6 : Data corruption error message not being sent to syslog

2022-05-25 Thread Abhishek Bhola
This morning I noticed this error in my PG CSV log file. ``` 2022-05-25 23:59:17.776 JST,,,117110,,628cf2c5.1c976,1,,2022-05-25 23:59:17 JST,9/611296,0,ERROR,XX001,"uncommitted xmin 16395356 from before xid cutoff 144683296 needs to be frozen","while scanning block 19267 of relation ""relation

Re: Why are data files getting modified?

2022-02-09 Thread Abhishek Bhola
I see, but then why is the autovacuum getting triggered after the select command? On Thu, Feb 10, 2022, 12:14 AM Laurenz Albe wrote: > On Wed, 2022-02-09 at 17:29 +0900, Abhishek Bhola wrote: > > I have a table with monthly partitions, starting from 2011. > > Each partitio

Why are data files getting modified?

2022-02-09 Thread Abhishek Bhola
I have a table with monthly partitions, starting from 2011. Each partition has about 2-3 billion rows and is about 40TB of data in total. I am running a select query to \copy the data into csv files for each month. I see there is an *AUTOVACUUM* job started on all the partitions that have been rea

Re: Subscription stuck at initialize state

2022-02-03 Thread Abhishek Bhola
Hi Vijaykumar Thank you so much for taking out so much of your time to recreate the bug. I checked the max_logical_replication_workers on both the nodes and they are set at 4 at the moment. The reason why it is failing is that there are 3 existing replications on the target node already and when

Re: Subscription stuck at initialize state

2022-02-03 Thread Abhishek Bhola
*3616584803 *| 3322052570 |* i * | 3616584803 | 3322052133 | *i * | * *pg_stat_subscription* targetdb=# select * from pg_stat_subscription ; subid|subname | pid | relid | received_lsn | last_msg_send_time | last_msg_receipt_time

Re: Subscription stuck at initialize state

2022-02-02 Thread Abhishek Bhola
Thu, Feb 3, 2022 at 3:33 PM Vijaykumar Jain < vijaykumarjain.git...@gmail.com> wrote: > > > On Thu, Feb 3, 2022, 10:32 AM Abhishek Bhola < > abhishek.bh...@japannext.co.jp> wrote: > >> So far I figured out that the problem is on the subscriber side. >> The s

Re: Subscription stuck at initialize state

2022-02-02 Thread Abhishek Bhola
:53 AM Abhishek Bhola < abhishek.bh...@japannext.co.jp> wrote: > Update: Tried dropping and recreating publication on the source DB and > subscription still won't move ahead. > Not sure what I am missing. > > On Wed, Feb 2, 2022 at 1:20 PM Abhishek Bhola < > abhis

Re: Subscription stuck at initialize state

2022-02-02 Thread Abhishek Bhola
Update: Tried dropping and recreating publication on the source DB and subscription still won't move ahead. Not sure what I am missing. On Wed, Feb 2, 2022 at 1:20 PM Abhishek Bhola < abhishek.bh...@japannext.co.jp> wrote: > The only statement I see on the target DB log is > &qu

Re: Subscription stuck at initialize state

2022-02-01 Thread Abhishek Bhola
blem blocking the subscription? For example, a subscribed > table missing a column that exists in the publisher. > > Cheers, > > Steve > > On Wed, Feb 2, 2022 at 1:26 PM Abhishek Bhola < > abhishek.bh...@japannext.co.jp> wrote: > >> I have 2 sets of publica

Subscription stuck at initialize state

2022-02-01 Thread Abhishek Bhola
I have 2 sets of publication/subscription between my 2 DBs. One of them is working fine and the other one is stuck at initializing state for all the tables. sourcedb=# select * from pg_publication; oid |pubname | pubowner | puballtables | pubinsert | pubupdate | pubdelete | pubt

Re: [Major version upgrade] pg_upgrade fails despite passing check mode

2021-10-24 Thread Abhishek Bhola
Thanks, let me try that out. But is there a way to know for sure if pg_upgrade will have any such problems? Other than FDW's owner, any other things I should check before upgrading other clusters? Regards Abhishek Bhola On Sat, Oct 23, 2021 at 12:20 AM Laurenz Albe wrote: > On Fri, 20

[Major version upgrade] pg_upgrade fails despite passing check mode

2021-10-22 Thread Abhishek Bhola
While upgrading from Postgres 11 to 13, the pg_upgrade passed the check mode postgres@dxxxpgs03:/D1/datadg/data13$ /usr/pgsql-13/bin/pg_upgrade -k --check --new-datadir='/D1/datadg/data13' --old-datadir='/D1/datadg/data' --new-bind$ r='/usr/pgsql-13/bin' --old-bindir='/usr/pgsql-11/bin

Re: Postgresql 11: terminating walsender process due to replication timeout

2021-09-10 Thread Abhishek Bhola
again and it stopped working. So given that the publisher is publishing at a higher rate than the subscriber is subscribing, what can be done? On Fri, Sep 10, 2021 at 9:26 AM Kyotaro Horiguchi wrote: > At Thu, 9 Sep 2021 16:06:25 +0900, Abhishek Bhola < > abhishek.bh...@japannext.c

Re: Postgresql 11: terminating walsender process due to replication timeout

2021-09-09 Thread Abhishek Bhola
sourcedb:~$ postgres --version postgres (PostgreSQL) 11.6 Sorry for missing this information. But looks like this fix is already included in the version I am running. Regards, Abhishek Bhola On Thu, Sep 9, 2021 at 3:56 PM Kyotaro Horiguchi wrote: > At Thu, 9 Sep 2021 14:52:25 +0900, Abhis

Postgresql 11: terminating walsender process due to replication timeout

2021-09-08 Thread Abhishek Bhola
I have found some questions about the same error, but didn't find any of them answering my problem. The setup is that I have two Postgres11 clusters (A and B) and they are making use of publication and subscription features to copy data from A to B. A (source DB- publication) --> B (t