RE: Are Foreign Key Disabled During Logical Replication Initial Sync?

2021-12-06 Thread Avi Weinberg
Hi Peter, Thanks for answer. Just to clarify, they are disabled during initial sync only or are always disabled on subscriber side? Are all triggers disabled during initial sync or just foreign keys? How can I know that initial sync completed for all tables? Is it checking when pg_subscription

Re: Advice on using materialized views

2021-12-06 Thread Nick Cleaton
On Mon, 6 Dec 2021 at 18:48, Phil Endecott wrote: > > - I have a raw data table, indexed on a timestamp column, to which > new data is regularly being added. > > - I have a set of views that produce hourly/daily/monthly summaries > of this data. Querying these views is slow, so I'd like to mak

Re: CTE Materialization

2021-12-06 Thread Paul van der Linden
It did indeed work as expected. Took the query down from over 18 hours to 20 minutes, so a huge win! Paul On Thu, Dec 2, 2021 at 11:34 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Thursday, December 2, 2021, Paul van der Linden < > paul.doskabou...@gmail.com> wrote: > >> Hi, >>

Re: Are Foreign Key Disabled During Logical Replication Initial Sync?

2021-12-06 Thread Peter Eisentraut
On 06.12.21 15:50, Avi Weinberg wrote: Does it mean that populating each table is done in a single transaction?  If so, when I have tables with foreign keys between them, is it guaranteed that logical replication will populates the tables in the proper order so the foreign key will be enforced?

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Mladen Gogala
On 12/6/21 22:16, Laurenz Albe wrote: An ugly alternative is to use "ORDER BY id + 0", which prevents PostgreSQL from using the index. That was actually the earliest form of Oracle hints. I remember doing exactly that in Oracle 5.1.22 on VAX/VMS. -- Mladen Gogala Database Consultant Tel: (34

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Laurenz Albe
On Mon, 2021-12-06 at 19:22 +0100, Pavel Stehule wrote: > po 6. 12. 2021 v 18:21 odesílatel Francisco Olarte > napsal: > > On Mon, 6 Dec 2021 at 18:03, Alan Hodgson wrote: > > > # explain SELECT "shipment_import_records".* FROM > > > "shipment_import_records" WHERE > > > "shipment_import_reco

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Rob Sargent
On 12/6/21 11:02 AM, Alan Hodgson wrote: On Mon, 2021-12-06 at 10:19 -0700, Rob Sargent wrote: To be clear, is it the devs or the ORM that's adding the ORDER  and the LIMIT?  I'm betting on devs.  Do they need the smallest id (first occurrance?) or do they need data common to all 5096 entries (N

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Pavel Stehule
po 6. 12. 2021 v 18:21 odesílatel Francisco Olarte napsal: > On Mon, 6 Dec 2021 at 18:03, Alan Hodgson > wrote: > ... > > The table has nearly 29 million records. 5069 of them match > shipment_import_id = 5090609. There is an index on shipment_import_id, > which the planner happily uses without

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Adrian Klaver
On 12/6/21 10:02, Alan Hodgson wrote: On Mon, 2021-12-06 at 10:19 -0700, Rob Sargent wrote: To be clear, is it the devs or the ORM that's adding the ORDER  and the LIMIT?  I'm betting on devs.  Do they need the smallest id (first occurrance?) or do they need data common to all 5096 entries (Name

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Alan Hodgson
On Mon, 2021-12-06 at 10:19 -0700, Rob Sargent wrote: > To be clear, is it the devs or the ORM that's adding the ORDER  and > the  > LIMIT?  I'm betting on devs.  Do they need the smallest id (first > occurrance?) or do they need data common to all 5096 entries > (Name?) and > any record will do?

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Alan Hodgson
On Mon, 2021-12-06 at 18:20 +0100, Francisco Olarte wrote: > > Can you post an explain analyze? To me it seems like the planner > thinks shipment_import_id is randomly distributed and the table is > well correlated with it's PK, so scanning it for the first id > should > be fast.    #explain anal

Re: Require details that how to find user creation date in postgresql Database

2021-12-06 Thread Sonai muthu raja M
Dear Tom, Thank you so much for sharing the information. Regards, M. Sonai Muthu Raja Managed Delivery Services - DBA Support Landline - 044 66235600, Extn - 1176​ Mobile - +91 9003132734 From: Tom Lane Sent: Friday, December 3, 2021 3:51 AM To: Sonai

Re: CTE Materialization

2021-12-06 Thread Paul van der Linden
Thanks a lot, completely forgot that one! Gonna test that tomorrow... On Thu, Dec 2, 2021 at 11:34 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Thursday, December 2, 2021, Paul van der Linden < > paul.doskabou...@gmail.com> wrote: > >> Hi, >> >> when switching to postgres 14 (fr

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Alan Hodgson
On Mon, 2021-12-06 at 10:18 -0700, Michael Lewis wrote: > What is your default_statistics_target and how accurate is that > estimate of 5668 rows? What is random_page_cost set to by the way? > > > default_statistics_target = 1000 random_page_cost = 2.0 (it's on AWS on a 9000 iops gp2 volume) P

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Francisco Olarte
On Mon, 6 Dec 2021 at 18:03, Alan Hodgson wrote: ... > The table has nearly 29 million records. 5069 of them match > shipment_import_id = 5090609. There is an index on shipment_import_id, which > the planner happily uses without the LIMIT specifically. Yet with it the > query planner will alway

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Rob Sargent
On 12/6/21 10:03 AM, Alan Hodgson wrote: I keep running into problems like these: Devs are using an ORM. It really likes to produce queries like: SELECT "shipment_import_records".* FROM "shipment_import_records" WHERE shipment_import_records"."shipment_import_id" = 5090609 ORDER BY "shipment_

Re: Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Michael Lewis
What is your default_statistics_target and how accurate is that estimate of 5668 rows? What is random_page_cost set to by the way? More importantly, what is the better plan that you'd like the planner to use with your existing indexes? It would seem logical to me to scan for the matching shipment_

Query planner issue with preferring primary key over a better index when using ORDER BY and LIMIT

2021-12-06 Thread Alan Hodgson
I keep running into problems like these: Devs are using an ORM. It really likes to produce queries like: SELECT "shipment_import_records".* FROM "shipment_import_records" WHERE shipment_import_records"."shipment_import_id" = 5090609 ORDER BY "shipment_import_records"."id" ASC LIMIT 1; I don't kn

Are Foreign Key Disabled During Logical Replication Initial Sync?

2021-12-06 Thread Avi Weinberg
Hi all, I understood that during logical replication initial sync tables are copied in full using some type of "copy command". Does it mean that populating each table is done in a single transaction? If so, when I have tables with foreign keys between them, is it guaranteed that logical replic

Re: libpq: Which functions may hang due to network issues?

2021-12-06 Thread Daniel Frey
On 5. Dec 2021, at 21:32, Daniel Frey wrote: > >> On 5. Dec 2021, at 17:01, Tom Lane wrote: >> >> Daniel Frey writes: >>> With all that said, I think that PostgreSQL/libpq should have a clear, >>> documented way to get rid of a connection that is guaranteed to not hang. >>> It has something