Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
fsync is off and full_page_writes is off because the script works one time. We create the db, we load the data, then we dump the data and kill the db. No need to handle servers crashed or anything like that. 0.1 vacuum_cleanup_index_scale_factor is the default value. On Thu, Jul 8, 2021 at 4:06 P

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
On a different machine, we use 12.7. Still same issue On Thu, Jul 8, 2021 at 3:49 PM Allan Barrielle wrote: > Hello, > > > Is it true that the SELECTs have no joins in them ? > > Yes there is a lot of LEFT JOIN. > > > When it doesn't work, you could check SELECT * FROM pg_stat_activity, > and >

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
Hello, > Is it true that the SELECTs have no joins in them ? Yes there is a lot of LEFT JOIN. > When it doesn't work, you could check SELECT * FROM pg_stat_activity, and >SELECT pg_blocking_pids(pid), * FROM pg_locks, to see what's going on. I can't see any blocking queries blocking pg_locks, p

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Justin Pryzby
On Thu, Jul 08, 2021 at 03:49:12PM +0200, Allan Barrielle wrote: > > Is it true that the SELECTs have no joins in them ? > > Yes there is a lot of LEFT JOIN. > > > It'd be very useful to get "explain analyze" for a working query and for a > > stuck query. It sound like the stuck query never fini

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Imre Samu
> We use different machines, different config, and different datasets. > ... > PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit Is It possible to upgrade and test with PG 12.7? IMHO: lot of changes: * https://www.postgresql.org/docs

Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Justin Pryzby
On Thu, Jul 08, 2021 at 01:00:28AM +0200, Allan Barrielle wrote: > All is fine, and can work great. > But sometimes, some queries that used to take about 20 secs to complete can > suddenly end in 5mins. > Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit > of transform) FR

ETL - sql orchestrator is stuck when there is not sleep() between queries

2021-07-08 Thread Allan Barrielle
Hello guys, I'm facing a problem. Currently I'm working on a Data transformation Pipeline on Postgres. The strategy is, We select every tables in a given schema ( 50 tables ), we apply some case when, translation, enum and load it into a different new schema with a CREATE TABLE SCHEMA_2.table_1 AS