Re: Long running query causing XID limit breach

2024-05-23 Thread sud
On Thu, May 23, 2024 at 8:11 PM Laurenz Albe wrote: > On Thu, 2024-05-23 at 18:15 +0530, sud wrote: > > On Thu, May 23, 2024 at 1:45 PM Laurenz Albe > wrote: > > > If a long running query on the standby influences the primary, that > means that > > > you have "hot_standby_feedback" set to "on".

Re: Strange issue with unique index

2024-05-23 Thread Tom Lane
writes: > I've run into a strange issue with a unique index that I'm struggling to > understand. I've extracted the basic info to reproduce this below. > ... > This will now block until session 2 is complete. I don't understand why this > would block. I do know it's that unique index causing the

Strange issue with unique index

2024-05-23 Thread rstander
Good day I've run into a strange issue with a unique index that I'm struggling to understand. I've extracted the basic info to reproduce this below. PG Version: 15.6 --Create structure CREATE SCHEMA IF NOT EXISTS playground; CREATE TABLE playground.parent ( parent_id bigint

Re: PG 12.2 ERROR: cannot freeze committed xmax

2024-05-23 Thread Ron Johnson
On Thu, May 23, 2024 at 9:41 AM bruno da silva wrote: > Hello, > I have a deployment with PG 12.2 reporting ERROR: cannot freeze committed > xmax > using Red Hat Enterprise Linux 8.9. > > What is the recommended to find any bug fixes that the version 12.2 had > that could have caused this error.

Re: expected authentication request from server, but received H

2024-05-23 Thread Tom Lane
"Peter J. Holzer" writes: > One of our users reports getting the error message > "expected authentication request from server, but received H" > when trying to connect to the database. That's very bizarre, and I don't recall any similar reports in the recent past. > Server is PostgreSQL 15.7

Need help to review a serializability implementation for MySQL Cluster

2024-05-23 Thread conflict_serializability
Hey guys, I've developed a serializability implementation for MySQL Cluster(NDB Cluster) and you are invited to peer-review it for me. I believe it is the fifth one in commercial database systems after: MySQL InnoDB's 2PL, PostgreSQL's Serializable Snapshot Isolation, Google's Spanner's

Re: Json table/column design question

2024-05-23 Thread David G. Johnston
On Thu, May 23, 2024, 10:07 Skorpeo Skorpeo wrote: > Thank you for the valuable feedback. I see people are big fans of json > here. > >> It's better for data interchange than as a basis for long term model storage. David J.

Re: Json table/column design question

2024-05-23 Thread Adrian Klaver
On 5/23/24 09:06, Skorpeo Skorpeo wrote: Thank you for the valuable feedback.  I see people are big fans of json here. You can be a fan of JSON and still think it is not the correct way to store data in a relational database. When you do that you often end up with nested, possibly

Re: Json table/column design question

2024-05-23 Thread Skorpeo Skorpeo
Thank you for the valuable feedback. I see people are big fans of json here. On Thu, May 23, 2024 at 3:04 AM Laurenz Albe wrote: > On Wed, 2024-05-22 at 22:38 -0500, Skorpeo Skorpeo wrote: > > I was wondering if having unrelated columns in a table is a sound > approach when > > using json. In

Re: Restore of a reference database kills the auto analyze processing.

2024-05-23 Thread Adrian Klaver
On 5/23/24 06:01, HORDER Philip wrote: Classified as: {OPEN} I actually hadn't thought to look for Postgres bugs, the system is so reliable for us, it just doesn't go wrong! I can trigger the fault be running a reload of the lfm database, and we've been running Postgres 10 & 13 for several

Re: Long running query causing XID limit breach

2024-05-23 Thread Laurenz Albe
On Thu, 2024-05-23 at 18:15 +0530, sud wrote: > On Thu, May 23, 2024 at 1:45 PM Laurenz Albe wrote: > > If a long running query on the standby influences the primary, that means > > that > > you have "hot_standby_feedback" set to "on".  Set it to "off". > > Will the setting up of

PG 12.2 ERROR: cannot freeze committed xmax

2024-05-23 Thread bruno da silva
Hello, I have a deployment with PG 12.2 reporting ERROR: cannot freeze committed xmax using Red Hat Enterprise Linux 8.9. What is the recommended to find any bug fixes that the version 12.2 had that could have caused this error. Could this error be caused by OS/Hardware related issues? Thanks --

Re: Restore of a reference database kills the auto analyze processing.

2024-05-23 Thread HORDER Philip
Classified as: {OPEN} I actually hadn't thought to look for Postgres bugs, the system is so reliable for us, it just doesn't go wrong! I can trigger the fault be running a reload of the lfm database, and we've been running Postgres 10 & 13 for several years now without seeing this problem.

Re: Long running query causing XID limit breach

2024-05-23 Thread sud
On Thu, May 23, 2024 at 1:45 PM Laurenz Albe wrote: > > > If a long running query on the standby influences the primary, that means > that > you have "hot_standby_feedback" set to "on". Set it to "off". > > > Will the setting up of "hot_standby_feedback" value to OFF will cause the reader

Re: Backup failure Postgres

2024-05-23 Thread Alvaro Herrera
On 2024-May-23, Jethish Jethish wrote: > I have tried by increasing the max_standby_streaming_delay but I'm facing > lag issues on the replica server. > > When i increase the max_standby_streaming_delay even if a query runs for 2 > minutes I'm facing lag issues for 2 minutes. You could use a

Re: Backup failure Postgres

2024-05-23 Thread Torsten Förtsch
Look, you have to compromise somewhere. Let me explain the problem. PG uses MVCC. That means if you update or delete rows, rows are not actually modified or added back to free space. They are just marked for later removal. That actual removal is VACUUM's task. The reason for doing so is that a

Re: Backup failure Postgres

2024-05-23 Thread Jethish Jethish
Hi Torsten, I have tried by increasing the max_standby_streaming_delay but I'm facing lag issues on the replica server. When i increase the max_standby_streaming_delay even if a query runs for 2 minutes I'm facing lag issues for 2 minutes. Please suggest here. Data size is 3TB On Thu, May 23,

Re: Backup failure Postgres

2024-05-23 Thread Torsten Förtsch
As the error message says, your query was aborted due to it conflicting with recovery. There are many ways to deal with that. You could enable hot_standby_feedback on the replica. You could disconnect the replica from the master for the time the COPY takes (reset primary_conninfo). You could

Backup failure Postgres

2024-05-23 Thread Jethish Jethish
I'm frequently facing the below error while performing backup. Someone please tell how solve this issues. Failed : pg_dump: error: Dumping the contents of table "botsession" failed: PQgetResult() failed. pg_dump: error: Error message from server: ERROR: canceling statement due to conflict with

Re: Long running query causing XID limit breach

2024-05-23 Thread Laurenz Albe
On Thu, 2024-05-23 at 13:41 +0530, sud wrote: > > Yes, that is correct.  You cannot run such long-running queries with a > > transaction rate like that. > > When you mean transaction ,does it mean one commit ? For example if it's > inserting+committing ~1000 rows in one batch then all the 1000

Re: Long running query causing XID limit breach

2024-05-23 Thread sud
On Thu, May 23, 2024 at 1:22 PM Laurenz Albe wrote: > On Thu, 2024-05-23 at 02:46 +0530, sud wrote: > > It's RDS postgres version 15.4. We suddenly saw the > "MaximumUsedTransactionIDs" > > reach to ~1.5billion and got alerted by team members who mentioned the > database > > is going to be in

Re: Json table/column design question

2024-05-23 Thread Laurenz Albe
On Wed, 2024-05-22 at 22:38 -0500, Skorpeo Skorpeo wrote: > I was wondering if having unrelated columns in a table is a sound approach > when > using json.  In other words, if I have two collections of unrelated json > objects, > for example "Users" and "Inventory", would it be ok to have one

Re: Long running query causing XID limit breach

2024-05-23 Thread Laurenz Albe
On Thu, 2024-05-23 at 02:46 +0530, sud wrote: > It's RDS postgres version 15.4. We suddenly saw the > "MaximumUsedTransactionIDs" > reach to ~1.5billion and got alerted by team members who mentioned the > database > is going to be in shutdown/hung if this value reaches to ~2billion and won't >

Re: Long running query causing XID limit breach

2024-05-23 Thread yudhi s
On Thu, May 23, 2024 at 11:42 AM sud wrote: > >> Calculation Rationale >> Daily XID Usage: Approximately 4 billion rows per day implies high XID >> consumption. >> Buffer Time: At 1 billion XIDs, you would still have 1 billion XIDs >> remaining, giving you roughly 12 hours to address the issue

expected authentication request from server, but received H

2024-05-23 Thread Peter J. Holzer
One of our users reports getting the error message "expected authentication request from server, but received H" when trying to connect to the database. I have confirmed that they do indeed connect to the database and not some other service and in Wireshark it looks like the TLS handshake

Re: Long running query causing XID limit breach

2024-05-23 Thread Muhammad Salahuddin Manzoor
Greetings, Yes, monitoring and alerting for VACUUM operations are crucial. Track VACUUM Duration and Success: SELECT pid, state, query_start, now() - query_start AS duration, query FROM pg_stat_activity WHERE query LIKE 'VACUUM%' ORDER BY duration DESC; Check Autovacuum Activity: SELECT

Re: Long running query causing XID limit breach

2024-05-23 Thread sud
Also,if i am getting it correct, it means we should not run any transaction (even if it's legitimate one like for e.g. a big Reporting "SELECT" query) beyond 10hrs, as that will end up consuming 10*200million XID per hour= 2billion XID limit saturation and thus causing system failure. Hope my

Re: Long running query causing XID limit breach

2024-05-23 Thread sud
On Thu, May 23, 2024 at 10:42 AM Muhammad Salahuddin Manzoor < salahuddi...@bitnine.net> wrote: > Greetings, > > Running `VACUUM table_name;` on a partitioned table will vacuum each > partition individually, not the whole table as a single unit. > > Yes, running `VACUUM table_name;` frequently on