Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-05-02 Thread Michael Paquier
On Sun, May 01, 2022 at 09:27:18PM -0700, Noah Misch wrote: > commit 322becb wrote: Thanks, Noah. I am out this week, but I should be able to address all your points at the beginning of next week. I have added an open item for now. -- Michael signature.asc Description: PGP signature

Add last failed connection error message to pg_stat_wal_receiver

2022-05-02 Thread Bharath Rupireddy
Hi, In production environments WAL receiver connection attempts to primary may fail for many reasons (primary down, network is broken, authentication tokens changes, primary_conn_info modifications, socket errors and so on.). Although we emit the error message to server logs, isn't it useful to sh

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-02 Thread Etsuro Fujita
Hi, On Wed, Apr 20, 2022 at 4:55 AM David Zhang wrote: > I tried to apply the patch to master and plan to run some tests, but got > below errors due to other commits. I rebased the patch against HEAD. Attached is an updated patch. > + * remote server in parallel at (sub)transaction end. >

Refactor construct_array() and deconstruct_array() for built-in types

2022-05-02 Thread Peter Eisentraut
[for PG16] There are many calls to construct_array() and deconstruct_array() for built-in types, for example, when dealing with system catalog columns. These all hardcode the type attributes necessary to pass to these functions. To simplify this a bit, add construct_array_builtin(), deconstr

strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
Hi I found a query that is significantly slower with more memory plan 1 - fast https://explain.depesz.com/s/XM1f plan 2 - slow https://explain.depesz.com/s/2rBw Strange - the time of last row is +/- same, but execution time is 10x worse It looks like slow environment cleaning

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 11:47 AM Masahiko Sawada wrote: > > On Fri, Apr 8, 2022 at 6:14 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, April 6, 2022 1:20 PM Amit Kapila > > wrote: > > > > > In this email, I would like to discuss allowing streaming logical > > > transactions (large in-p

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 07:31, Amit Kapila wrote: > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra > wrote: >> >> On 4/30/22 12:11, Amit Kapila wrote: >>> On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera >>> wrote: My proposal is that if users want to define multiple publications, and their defin

Add missing MarkGUCPrefixReserved() in basebackup_to_shell module

2022-05-02 Thread Bharath Rupireddy
Hi, It looks like commit c6306db24 (Add 'basebackup_to_shell' contrib module.) missed to reserve basebackup_to_shell module's custom GUC prefix via MarkGUCPrefixReserved("basebackup_to_shell");. This will remove Attaching a tiny patch to fix it. Regards, Bharath Rupireddy. v1-0001-Add-missing-

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > On 5/2/22 07:31, Amit Kapila wrote: > > Yeah, or don't allow to define such publications in the first place so > > that different subscriptions can't combine them but I guess that might > > forbid some useful cases as well where publication may not get > > com

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 12:17, Alvaro Herrera wrote: > On 2022-May-02, Tomas Vondra wrote: >> On 5/2/22 07:31, Amit Kapila wrote: > >>> Yeah, or don't allow to define such publications in the first place so >>> that different subscriptions can't combine them but I guess that might >>> forbid some useful case

Re: avoid multiple hard links to same WAL file after a crash

2022-05-02 Thread Michael Paquier
On Sun, May 01, 2022 at 10:08:53PM +0900, Michael Paquier wrote: > Now, I am surprised by the third code path of durable_rename_excl(), > as of the WAL receiver doing writeTimeLineHistoryFile(), to not cause > any issues, as link() should exit with EEXIST when the startup process > grabs the same h

Re: Add missing MarkGUCPrefixReserved() in basebackup_to_shell module

2022-05-02 Thread Michael Paquier
On Mon, May 02, 2022 at 03:06:39PM +0530, Bharath Rupireddy wrote: > It looks like commit c6306db24 (Add 'basebackup_to_shell' contrib > module.) missed to reserve basebackup_to_shell module's custom GUC > prefix via MarkGUCPrefixReserved("basebackup_to_shell");. This will > remove > > Attaching a

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > On 5/2/22 12:17, Alvaro Herrera wrote: > > The latter ultimately means that we aren't sure that a combined > > subscription is safe. And in turn this means that a pg_dump of such a > > database cannot be restored (because the CREATE SUBSCRIPTION will be > >

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 3:53 PM Tomas Vondra wrote: > > On 5/2/22 12:17, Alvaro Herrera wrote: > > On 2022-May-02, Tomas Vondra wrote: > >> On 5/2/22 07:31, Amit Kapila wrote: > > > >>> Yeah, or don't allow to define such publications in the first place so > >>> that different subscriptions can't c

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 3:05 PM Tomas Vondra wrote: > > On 5/2/22 07:31, Amit Kapila wrote: > > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra > > wrote: > >> > > >> The second option has the annoying consequence that it makes this > >> useless for the "data redaction" use case I described in [2], be

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-02 Thread Masahiko Sawada
On Mon, May 2, 2022 at 6:09 PM Amit Kapila wrote: > > On Mon, May 2, 2022 at 11:47 AM Masahiko Sawada wrote: > > > > On Fri, Apr 8, 2022 at 6:14 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Wednesday, April 6, 2022 1:20 PM Amit Kapila > > > wrote: > > > > > > > In this email, I would li

Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Bharath Rupireddy
Hi, Currently the emit_log_hook gets called only for the log messages of type <= log_min_message i.e when edata->output_to_server is true [1], which means that I can't use an implementation of emit_log_hook to just intercept, say, all DEBUGX messages without interrupting the actual server logs flo

Re: simplifying foreign key/RI checks

2022-05-02 Thread Amit Langote
On Mon, Apr 11, 2022 at 4:47 PM Amit Langote wrote: > This one has been marked Returned with Feedback in the CF app, which > makes sense given the discussion on -committers [1]. > > Agree with the feedback given that it would be better to address *all* > RI trigger check/action functions in the pr

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Amit Kapila wrote: > We don't do that currently but we can as mentioned in my previous > email [1]. Let me write the relevant part again. We need to expose all > publications for a walsender, and then we can find the exact set of > publications where the current publication is used

limiting collected query text length in pg_stat_statements

2022-05-02 Thread Godfrin, Philippe E
Greetings, I want to limit the query text that gets captured in pg_stat_statements. We have sql statements with thousands of values clauses (upwards of 10,000) that run at a 1 second interval. When just a handful are running plus 2 or 3 loads using the same technique (10,000 entry values clause

Item compression in the Gist index

2022-05-02 Thread Baca Radim
Hi, we are developing an extension for multidimensional data. We have created a Gist index that is heavily inspired by a cube extension. Right now we would like to add some item compression since data items in a node share a significant portion of a tuple prefix. However, I have no idea how to

Add pg_strtoupper and pg_strtolower functions

2022-05-02 Thread Bharath Rupireddy
Hi, I came across pg_toupper and pg_tolower functions, converting a single character, are being used in loops to convert an entire null-terminated string. The cost of calling these character-based conversion functions (even though small) can be avoided if we have two new functions pg_strtoupper an

Re: Progress report removal of temp files and temp relation files using ereport_startup_progress

2022-05-02 Thread Ashutosh Bapat
Hi Bharath, On Sat, Apr 30, 2022 at 11:08 AM Bharath Rupireddy wrote: > > Hi, > > At times, there can be many temp files (under pgsql_tmp) and temp > relation files (under removal which after crash may take longer during > which users have no clue about what's going on in the server before it >

Re: Item compression in the Gist index

2022-05-02 Thread Laurenz Albe
On Mon, 2022-05-02 at 14:48 +0200, Baca Radim wrote: > we are developing an extension for multidimensional data. We have > created a Gist index that is heavily inspired by a cube extension. Right > now we would like to add some item compression since data items in a > node share a significant po

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Julien Rouhaud
Hi, On Mon, May 02, 2022 at 05:11:34PM +0530, Bharath Rupireddy wrote: > > Currently the emit_log_hook gets called only for the log messages of > type <= log_min_message i.e when edata->output_to_server is true [1], > which means that I can't use an implementation of emit_log_hook to > just interc

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Bharath Rupireddy
On Mon, May 2, 2022 at 6:32 PM Julien Rouhaud wrote: > > Hi, > > On Mon, May 02, 2022 at 05:11:34PM +0530, Bharath Rupireddy wrote: > > > > Currently the emit_log_hook gets called only for the log messages of > > type <= log_min_message i.e when edata->output_to_server is true [1], > > which means

Re: Add pg_strtoupper and pg_strtolower functions

2022-05-02 Thread Ashutosh Bapat
On Mon, May 2, 2022 at 6:21 PM Bharath Rupireddy wrote: > > Hi, > > I came across pg_toupper and pg_tolower functions, converting a single > character, are being used in loops to convert an entire > null-terminated string. The cost of calling these character-based > conversion functions (even thou

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Julien Rouhaud
On Mon, May 02, 2022 at 06:40:05PM +0530, Bharath Rupireddy wrote: > On Mon, May 2, 2022 at 6:32 PM Julien Rouhaud wrote: > > > > Unless I'm missing something you can already do all of that with the current > > hook, since as mentioned in the comment above the hook can disable the > > server's >

Re: testclient.exe installed under MSVC

2022-05-02 Thread Daniel Gustafsson
> On 1 May 2022, at 15:23, Michael Paquier wrote: > > On Sun, May 01, 2022 at 01:07:06AM -0700, Noah Misch wrote: >> My annual audit for executables missing Windows icons turned up these: >> >>pginstall/bin/testclient.exe >>pginstall/bin/uri-regress.exe >> >> I was going to add

configure openldap crash warning

2022-05-02 Thread Peter Eisentraut
configure can report this: configure: WARNING: *** With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, each backend *** process that loads libpq (via WAL receiver, dblink, or postgres_fdw) and *** also uses LDAP will crash on exit. The source code also says # PostgreSQL sometimes loads lib

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 11:00, Pavel Stehule wrote: > > Hi > > I found a query that is significantly slower with more memory Which PostgreSQL version did you use? Did you enable assert checking? Do you have an example database setup to work with? > plan 2 > QUERY PLAN > > Nested

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
Enclosed is v3 of this patch; this adds two modes for this feature, one with the raw page `--save-fullpage/-W` and one with the LSN+checksum fixups `--save-fullpage-fixup/-X`. I've added at least some basic sanity-checking of the underlying feature, as well as run the test file and the changes to

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Bharath Rupireddy
On Mon, May 2, 2022 at 6:44 PM Julien Rouhaud wrote: > > On Mon, May 02, 2022 at 06:40:05PM +0530, Bharath Rupireddy wrote: > > On Mon, May 2, 2022 at 6:32 PM Julien Rouhaud wrote: > > > > > > Unless I'm missing something you can already do all of that with the > > > current > > > hook, since as

Re: limiting collected query text length in pg_stat_statements

2022-05-02 Thread Julien Rouhaud
Hi, On Mon, May 02, 2022 at 12:45:28PM +, Godfrin, Philippe E wrote: > Greetings, > > I want to limit the query text that gets captured in pg_stat_statements. We > have sql statements with thousands of values clauses (upwards of 10,000) that > run at a 1 second interval. When just a handful a

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
po 2. 5. 2022 v 15:28 odesílatel Matthias van de Meent < boekewurm+postg...@gmail.com> napsal: > On Mon, 2 May 2022 at 11:00, Pavel Stehule > wrote: > > > > Hi > > > > I found a query that is significantly slower with more memory > > Which PostgreSQL version did you use? Did you enable assert che

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Julien Rouhaud
On Mon, May 02, 2022 at 07:24:04PM +0530, Bharath Rupireddy wrote: > On Mon, May 2, 2022 at 6:44 PM Julien Rouhaud wrote: > > > > I know. What I said you could do is configure log_min_message to DEBUGX, so > > your extension sees everything you want it to see. And *in your extension* > > set >

Re: configure openldap crash warning

2022-05-02 Thread Tom Lane
Peter Eisentraut writes: > configure can report this: > configure: WARNING: > *** With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, each backend > *** process that loads libpq (via WAL receiver, dblink, or postgres_fdw) and > *** also uses LDAP will crash on exit. > I wonder whether we can

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-02 Thread Tom Lane
Julien Rouhaud writes: > On Mon, May 02, 2022 at 07:24:04PM +0530, Bharath Rupireddy wrote: >> I basically want to avoid normal users/developers setting any >> parameter (especially the superuser-only log_min_message GUC, all >> users might not have superuser access in production environments) or

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule napsal: > > > po 2. 5. 2022 v 15:28 odesílatel Matthias van de Meent < > boekewurm+postg...@gmail.com> napsal: > >> On Mon, 2 May 2022 at 11:00, Pavel Stehule >> wrote: >> > >> > Hi >> > >> > I found a query that is significantly slower with more me

Re: DBT-5 Stored Procedure Development (2022)

2022-05-02 Thread Mark Wong
On Mon, May 02, 2022 at 07:14:28AM -0700, Mark Wong wrote: > On Tue, Apr 26, 2022, 10:45 AM Peter Geoghegan wrote: > > > On Tue, Apr 26, 2022 at 10:36 AM Mark Wong wrote: > > > I'm afraid not. I'm guessing that pulling in egen 1.14 would address > > > that. Maybe it would make sense to put tha

Configuration Parameter/GUC value validation hook

2022-05-02 Thread Bharath Rupireddy
Hi, Right now postgres can't prevent users setting certain configuration parameters or GUCs (like shared_buffers, temp_buffers, work_mem, maintenance_work_mem, max_stack_depth, temp_file_limit, max_worker_processes, other worker processes settings, effective_io_concurrency and so on) to unreasonab

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 16:09, Pavel Stehule wrote: > > > > po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule > napsal: >> there is just shared buffers changed to 32GB and work_mem to 70MB. >> Unfortunately - it is in production environment with customer data, so I >> cannot to play too much >> >>

Re: [RFC] building postgres with meson -v8

2022-05-02 Thread Peter Eisentraut
On 29.04.22 19:46, Andres Freund wrote: explicitly using shared_library() rather than library() Why is that? We do build static libraries right now, so using library() would seem more suitable for that.

Re: Refactor construct_array() and deconstruct_array() for built-in types

2022-05-02 Thread Tom Lane
Peter Eisentraut writes: > There are many calls to construct_array() and deconstruct_array() for > built-in types, for example, when dealing with system catalog columns. > These all hardcode the type attributes necessary to pass to these functions. > To simplify this a bit, add construct_array_

Re: Proposal for internal Numeric to Uint64 conversion function.

2022-05-02 Thread Peter Eisentraut
On 22.04.22 14:26, Amul Sul wrote: Yes, I think we can do cleanup to some extent. Attaching the following patches that first intend to remove DirectFunctionCall as much as possible: Do you have any data that supports removing DirectionFunctionCall() invocations? I suppose some performance be

Re: Configuration Parameter/GUC value validation hook

2022-05-02 Thread Tom Lane
Bharath Rupireddy writes: > I propose to add a simple new hook in set_config_option (void > set_config_option_hook(struct config_generic *record);) and the > vendors can implement their own platform-dependent extensions to > accept or reject certain parameters (based on platform/VM > configuration

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
po 2. 5. 2022 v 16:44 odesílatel Matthias van de Meent < boekewurm+postg...@gmail.com> napsal: > On Mon, 2 May 2022 at 16:09, Pavel Stehule > wrote: > > > > > > > > po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule > napsal: > >> there is just shared buffers changed to 32GB and work_mem to 70MB. >

Re: testclient.exe installed under MSVC

2022-05-02 Thread Jacob Champion
On Mon, 2022-05-02 at 15:14 +0200, Daniel Gustafsson wrote: > Using a pg_ prefix > makes them sound like actual useful tools though with (albeit small) risk for > confusion? Noah's suggestion of libpq_ is perhaps better: libpq_testclient. +1 I also like Justin's idea of only installing the test

Re: Hash index build performance tweak from sorting

2022-05-02 Thread Simon Riggs
On Sat, 30 Apr 2022 at 12:12, Amit Kapila wrote: > > On Tue, Apr 19, 2022 at 3:05 AM Simon Riggs > wrote: > > > > Hash index pages are stored in sorted order, but we don't prepare the > > data correctly. > > > > We sort the data as the first step of a hash index build, but we > > forget to sort t

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-Apr-28, Tomas Vondra wrote: > SELECT > (CASE WHEN (a < 0) OR (a > 0) THEN a ELSE NULL END) AS a, > (CASE WHEN (a > 0) THEN b ELSE NULL END) AS b, > (CASE WHEN (a < 0) THEN c ELSE NULL END) AS c > FROM uno WHERE (a < 0) OR (a > 0) BTW, looking at the new COPY commands, the

Re: [RFC] building postgres with meson -v8

2022-05-02 Thread Andres Freund
Hi, On 2022-05-02 16:47:43 +0200, Peter Eisentraut wrote: > On 29.04.22 19:46, Andres Freund wrote: > > explicitly using shared_library() rather than library() > > Why is that? We do build static libraries right now, so using library() > would seem more suitable for that. When I wrote this I ha

Re: allow specifying action when standby encounters incompatible parameter settings

2022-05-02 Thread Nathan Bossart
On Fri, Apr 29, 2022 at 06:35:52PM +, Cary Huang wrote: > The patch applied and tested fine. I think for this kind of exception, it > really is up to the administrator how he/she should proceed to resolve > depending on his/her business application. Leaving things configurable by the > user

Re: Add --{no-,}bypassrls flags to createuser

2022-05-02 Thread Nathan Bossart
On Thu, Apr 28, 2022 at 03:06:30PM +0900, Shinya Kato wrote: > On 2022-04-26 05:19, Nathan Bossart wrote: >> +1 for "member". It might not be perfect, but IMO it's the clearest >> option. > > Thanks! I changed the option "--membership" to "--member". Thanks for the new patch! Would you mind add

Odd LOG output from "postgres -C"

2022-05-02 Thread Bruce Momjian
I just tested "postgres -C" on Postgres head, and got odd LOG output lines: $ postgres -C shared_memory_size 143 --> 2022-05-02 13:08:06.445 EDT [1582048] LOG: database system is shut down $ postgres -C "wal_segment_size" 16777216 --> 2022-05-02 13:13:30.4

Re: Odd LOG output from "postgres -C"

2022-05-02 Thread Nathan Bossart
On Mon, May 02, 2022 at 01:15:00PM -0400, Bruce Momjian wrote: > I just tested "postgres -C" on Postgres head, and got odd LOG output > lines: > > $ postgres -C shared_memory_size > 143 > --> 2022-05-02 13:08:06.445 EDT [1582048] LOG: database system is shut down > > $ postgr

Re: Odd LOG output from "postgres -C"

2022-05-02 Thread Bruce Momjian
On Mon, May 2, 2022 at 10:21:30AM -0700, Nathan Bossart wrote: > On Mon, May 02, 2022 at 01:15:00PM -0400, Bruce Momjian wrote: > > I just tested "postgres -C" on Postgres head, and got odd LOG output > > lines: > > > > $ postgres -C shared_memory_size > > 143 > > --> 2022-05-02 13:08:06.

Re: avoid multiple hard links to same WAL file after a crash

2022-05-02 Thread Nathan Bossart
On Sun, May 01, 2022 at 10:08:53PM +0900, Michael Paquier wrote: > At the end, switching directly from durable_rename_excl() to > durable_rename() should be fine for the WAL segment initialization, > but we could do things a bit more carefully by adding a check on the > file existence before callin

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 13:44, Alvaro Herrera wrote: > On 2022-May-02, Amit Kapila wrote: > >> We don't do that currently but we can as mentioned in my previous >> email [1]. Let me write the relevant part again. We need to expose all >> publications for a walsender, and then we can find the exact set of >>

Re: avoid multiple hard links to same WAL file after a crash

2022-05-02 Thread Nathan Bossart
On Mon, May 02, 2022 at 07:48:18PM +0900, Michael Paquier wrote: > Skimming through at the buildfarm logs, it happens that the tests are > able to see this race from time to time. Here is one such example on > rorqual: > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=rorqual&dt=2022

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > pgoutput.c is relies on relcache callbacks to get notified of changes. > See the stuff that touches replicate_valid and publications_valid. So > the walsender should notice the changes immediately. Hmm, I suppose that makes any changes easy enough to detect.

Is Dynamic Tracing in Postgres running on Ubuntu a good choice?

2022-05-02 Thread Bharath Rupireddy
Hi, I'm exploring the Dynamic Tracing [1] facility that postgres provides and planning to set it up in a reliable way on postgres running on Ubuntu. It looks like SystemTap is available on Ubuntu whereas DTrace isn't. I have no experience in using any of these tools. I would like to hear from hack

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 13:23, Amit Kapila wrote: > On Mon, May 2, 2022 at 3:05 PM Tomas Vondra > wrote: >> >> On 5/2/22 07:31, Amit Kapila wrote: >>> On Mon, May 2, 2022 at 3:27 AM Tomas Vondra >>> wrote: >> The second option has the annoying consequence that it makes this useless for the "d

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 19:51, Alvaro Herrera wrote: > On 2022-May-02, Tomas Vondra wrote: > >> pgoutput.c is relies on relcache callbacks to get notified of changes. >> See the stuff that touches replicate_valid and publications_valid. So >> the walsender should notice the changes immediately. > > Hmm, I s

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Peter Eisentraut
On 01.05.22 23:42, Tomas Vondra wrote: Imagine have a table with customers from different regions, and you want to replicate the data somewhere else, but for some reason you can only replicate details for one particular region, and subset of columns for everyone else. So you'd do something like t

Re: fix cost subqueryscan wrong parallel cost

2022-05-02 Thread Robert Haas
On Fri, Apr 29, 2022 at 3:38 PM Tom Lane wrote: > I wrote: > > So perhaps we should do it more like the attached, which produces > > this plan for the UNION case: > > sigh ... actually attached this time. I am not sure whether this is actually correct, but it seems a lot more believable than the

Re: fix cost subqueryscan wrong parallel cost

2022-05-02 Thread Tom Lane
Robert Haas writes: > I am not sure whether this is actually correct, but it seems a lot > more believable than the previous proposals. The problem might be more > general, though. I think when I developed this parallel query stuff I > modeled a lot of it on what you did for parameterized paths. B

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David Rowley
On Mon, 2 May 2022 at 21:00, Pavel Stehule wrote: > I found a query that is significantly slower with more memory Can you clarify what you mean here? More memory was installed on the machine? or work_mem was increased? or? > plan 1 - fast https://explain.depesz.com/s/XM1f > > plan 2 - slow http

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Tom Lane
David Rowley writes: > On Mon, 2 May 2022 at 21:00, Pavel Stehule wrote: >> I found a query that is significantly slower with more memory > If it was work_mem you increased, it seems strange that the plan would > switch over to using a Nested Loop / Memoize plan. Yeah, there's something unexpla

Re: avoid multiple hard links to same WAL file after a crash

2022-05-02 Thread Nathan Bossart
On Mon, May 02, 2022 at 10:39:07AM -0700, Nathan Bossart wrote: > On Mon, May 02, 2022 at 07:48:18PM +0900, Michael Paquier wrote: >> The WAL receiver upgrades the ERROR to a FATAL, and restarts >> streaming shortly after. Using durable_rename() would not be an issue >> here. > > Thanks for inves

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
...and pushing a couple fixups pointed out by cfbot, so here's v4. On Mon, May 2, 2022 at 8:42 AM David Christensen wrote: > > Enclosed is v3 of this patch; this adds two modes for this feature, > one with the raw page `--save-fullpage/-W` and one with the > LSN+checksum fixups `--save-fullpage-

PostgreSQL 15 Beta 1 release date

2022-05-02 Thread Jonathan S. Katz
Hi, The release date for PostgreSQL 15 Beta 1 is scheduled for May 19, 2022. Please ensure you have committed any work for Beta 1 by May 15, 2022 AoE[1]. Thank you for your efforts with resolving open items[2] as we work to stabilize PostgreSQL 15 for GA! Thanks, Jonathan [1]https://en.wik

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David G. Johnston
On Mon, May 2, 2022 at 4:02 PM Tom Lane wrote: > David Rowley writes: > > On Mon, 2 May 2022 at 21:00, Pavel Stehule > wrote: > >> I found a query that is significantly slower with more memory > > > If it was work_mem you increased, it seems strange that the plan would > > switch over to using

Re: failures in t/031_recovery_conflict.pl on CI

2022-05-02 Thread Andres Freund
Hi, On 2022-04-29 19:26:59 -0400, Tom Lane wrote: > Andres Freund writes: > > - The test uses pump_until() and wait_for_log(), which don't exist in the > > backbranches. For now I've just inlined the implementation, but I guess we > > could also backpatch their introduction? > > I'd backpatc

Re: testclient.exe installed under MSVC

2022-05-02 Thread Noah Misch
On Mon, May 02, 2022 at 03:14:50PM +0200, Daniel Gustafsson wrote: > > On 1 May 2022, at 15:23, Michael Paquier wrote: > > On Sun, May 01, 2022 at 01:07:06AM -0700, Noah Misch wrote: > >> My annual audit for executables missing Windows icons turned up these: > >> > >>pginstall/bin/testcli

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David Rowley
On Tue, 3 May 2022 at 11:02, Tom Lane wrote: > > David Rowley writes: > > On Mon, 2 May 2022 at 21:00, Pavel Stehule wrote: > >> I found a query that is significantly slower with more memory > > > If it was work_mem you increased, it seems strange that the plan would > > switch over to using a N

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David Rowley
On Tue, 3 May 2022 at 13:43, David G. Johnston wrote: > hit_ratio = (est_entries / ndistinct) - (ndistinct / calls) || clamp to 0.0 > I don't understand the adjustment factor ndistinct/calls I've attached a spreadsheet showing you the impact of subtracting (ndistinct / calls). What this is corre

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David G. Johnston
On Mon, May 2, 2022 at 7:13 PM David Rowley wrote: > On Tue, 3 May 2022 at 11:02, Tom Lane wrote: > > > > David Rowley writes: > > > On Mon, 2 May 2022 at 21:00, Pavel Stehule > wrote: > > >> I found a query that is significantly slower with more memory > > > > > If it was work_mem you increas

mylodon's failures in the back branches

2022-05-02 Thread Tom Lane
The reason that mylodon has been failing in v10 and v11 for awhile is that "-Werror=c99-extensions" breaks its test for : configure:12708: checking for stdbool.h that conforms to C99 configure:12775: ccache clang-13 -c -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wend

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David G. Johnston
On Mon, May 2, 2022 at 7:30 PM David Rowley wrote: > On Tue, 3 May 2022 at 13:43, David G. Johnston > wrote: > > hit_ratio = (est_entries / ndistinct) - (ndistinct / calls) || clamp to > 0.0 > > I don't understand the adjustment factor ndistinct/calls > > I've attached a spreadsheet showing you

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Tue, May 3, 2022 at 12:10 AM Tomas Vondra wrote: > > On 5/2/22 19:51, Alvaro Herrera wrote: > >> Why would we need to know publications replicated by other walsenders? > >> And what if the subscriber is not connected at the moment? In that case > >> there'll be no walsender. > > > > Sure, if th

Re: failures in t/031_recovery_conflict.pl on CI

2022-05-02 Thread Tom Lane
Andres Freund writes: > I ended up committing the extension of the test first, before the fix. I think > that's the cause of the failure on longfin on serinus. Let's hope the > situation improves with the now also committed (and backpatched) fix. longfin's definitely not very happy: four out of s

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera wrote: > > On 2022-May-02, Amit Kapila wrote: > > > I think it is possible to expose a list of publications for each > > walsender as it is stored in each walsenders > > LogicalDecodingContext->output_plugin_private. AFAIK, each walsender > > can have

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread David Rowley
On Tue, 3 May 2022 at 15:22, David G. Johnston wrote: > Plugging in: > n = 2,000 > e = 500 > c = 10,000 > > proper = 5% > incorrect = 25% > > But of the 10,000 calls we will receive, the first 2,000 will be misses while > 2,000 of the remaining 8,000 will be hits, due to sharing 2,000 distinct >

Re: failures in t/031_recovery_conflict.pl on CI

2022-05-02 Thread Andres Freund
Hi, On 2022-05-02 23:44:32 -0400, Tom Lane wrote: > Andres Freund writes: > > I ended up committing the extension of the test first, before the fix. I > > think > > that's the cause of the failure on longfin on serinus. Let's hope the > > situation improves with the now also committed (and backp

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
po 2. 5. 2022 v 23:48 odesílatel David Rowley napsal: > On Mon, 2 May 2022 at 21:00, Pavel Stehule > wrote: > > I found a query that is significantly slower with more memory > > Can you clarify what you mean here? More memory was installed on the > machine? or work_mem was increased? or? > > >

Re: mylodon's failures in the back branches

2022-05-02 Thread Andres Freund
Hi, On 2022-05-02 23:18:20 -0400, Tom Lane wrote: > The reason that mylodon has been failing in v10 and v11 for awhile > is that "-Werror=c99-extensions" breaks its test for : Was planning to send an email once I looked into it in a bit more detail... > configure:12708: checking for stdbool.h t

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 5:06 PM Masahiko Sawada wrote: > > On Mon, May 2, 2022 at 6:09 PM Amit Kapila wrote: > > > > On Mon, May 2, 2022 at 11:47 AM Masahiko Sawada > > wrote: > > > > > > > > > Are you planning to support "Transaction dependency" Amit mentioned in > > > his first mail in this pa

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
út 3. 5. 2022 v 6:09 odesílatel Pavel Stehule napsal: > > > po 2. 5. 2022 v 23:48 odesílatel David Rowley > napsal: > >> On Mon, 2 May 2022 at 21:00, Pavel Stehule >> wrote: >> > I found a query that is significantly slower with more memory >> >> Can you clarify what you mean here? More memory

Re: mylodon's failures in the back branches

2022-05-02 Thread Tom Lane
Andres Freund writes: > On 2022-05-02 23:18:20 -0400, Tom Lane wrote: >> which causes us to not use stdbool.h, which might be all right if you >> weren't also specifying --with-icu. > How did you conclude that ICU is the problem? I didn't immediately find > anything in the buildfarm output indica

Re: mylodon's failures in the back branches

2022-05-02 Thread Andres Freund
Hi, On 2022-05-03 00:24:09 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2022-05-02 23:18:20 -0400, Tom Lane wrote: > > Seems easiest to just change the configuration so that ICU isn't enabled for > > 10, 11? It's pretty reasonable to rely on it these days... > > Yeah, that seemed like t

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Tom Lane
Pavel Stehule writes: > there is really something strange (see attached file). Looks so this issue > is much more related to planning time than execution time You sure there's not something taking an exclusive lock on one of these tables every so often? regards, tom lane

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Pavel Stehule
út 3. 5. 2022 v 6:57 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > there is really something strange (see attached file). Looks so this > issue > > is much more related to planning time than execution time > > You sure there's not something taking an exclusive lock on one of these > ta

Re: failures in t/031_recovery_conflict.pl on CI

2022-05-02 Thread Tom Lane
Andres Freund writes: > On 2022-05-02 23:44:32 -0400, Tom Lane wrote: >> I can poke into that tomorrow, but are you sure that that isn't an >> expectable result? > It's not expected. But I think I might see what the problem is: > We wait for the FETCH (and thus the buffer pin to be acquired). But

Re: Proposal for internal Numeric to Uint64 conversion function.

2022-05-02 Thread Amul Sul
On Mon, May 2, 2022 at 8:23 PM Peter Eisentraut wrote: > > On 22.04.22 14:26, Amul Sul wrote: > > Yes, I think we can do cleanup to some extent. Attaching the > > following patches that first intend to remove DirectFunctionCall as > > much as possible: > > Do you have any data that supports remov