Re: new heapcheck contrib module

2020-07-20 Thread Amul Sul
On Tue, Jul 21, 2020 at 10:58 AM Amul Sul wrote: > > Hi Mark, > > I think new structures should be listed in src/tools/pgindent/typedefs.list, > otherwise, pgindent might disturb its indentation. > > Regards, > Amul > > > On Tue, Jul 21, 2020 at 2:32 AM Mark Dilger > wrote: > > > > > > > > > On J

Re: OpenSSL 3.0.0 compatibility

2020-07-20 Thread Michael Paquier
On Sun, Jul 19, 2020 at 04:29:54PM +0200, Peter Eisentraut wrote: > Good point. I have committed it with that adjustment. Also, I had the > format of the version number wrong, so I changed that, too. Thanks. I was not paying much attention to the format, but what you have committed is in line w

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Dilip Kumar
On Tue, Jul 21, 2020 at 2:00 AM Andres Freund wrote: > > Hi, > > On 2020-07-17 16:16:23 +0530, Dilip Kumar wrote: > > The attached patch allows the vacuum to continue by emitting WARNING > > for the corrupted tuple instead of immediately error out as discussed > > at [1]. > > > > Basically, it pro

Re: new heapcheck contrib module

2020-07-20 Thread Amul Sul
Hi Mark, I think new structures should be listed in src/tools/pgindent/typedefs.list, otherwise, pgindent might disturb its indentation. Regards, Amul On Tue, Jul 21, 2020 at 2:32 AM Mark Dilger wrote: > > > > > On Jul 16, 2020, at 12:38 PM, Robert Haas wrote: > > > > On Mon, Jul 6, 2020 at 2

Re: expose parallel leader in CSV and log_line_prefix

2020-07-20 Thread Michael Paquier
On Fri, Jul 17, 2020 at 07:34:54AM +0200, Julien Rouhaud wrote: > I had the same concern and was thinking about this approach too. > Another argument is that IIUC any log emitted due to > log_min_duration_statement wouldn't see the backend as executing a > parallel query, since the workers would al

Re: expose parallel leader in CSV and log_line_prefix

2020-07-20 Thread Michael Paquier
On Mon, Jul 20, 2020 at 11:12:31PM -0500, Justin Pryzby wrote: > On Tue, Jul 21, 2020 at 12:51:45PM +0900, Michael Paquier wrote: > The documentation could talk about either: > > 1) "lock group leader" - low-level, raw view of the internal data structure > (with a secondary mention that "for a par

Re: shared tempfile was not removed on statement_timeout

2020-07-20 Thread Justin Pryzby
On Fri, Dec 13, 2019 at 03:03:47PM +1300, Thomas Munro wrote: > On Fri, Dec 13, 2019 at 7:05 AM Justin Pryzby wrote: > > I have a nagios check on ancient tempfiles, intended to catch debris left by > > crashed processes. But triggered on this file: > > > > $ sudo find /var/lib/pgsql/12/data/base/

Re: expose parallel leader in CSV and log_line_prefix

2020-07-20 Thread Justin Pryzby
On Tue, Jul 21, 2020 at 12:51:45PM +0900, Michael Paquier wrote: > And I'd rather keep the simple suggestion of upthread to leave the > field as NULL for the parallel group leader with a PID match but not a > backend type check so as this could be useful for other types of > processes. The documen

Re: expose parallel leader in CSV and log_line_prefix

2020-07-20 Thread Michael Paquier
On Mon, Jul 20, 2020 at 06:30:48PM -0500, Justin Pryzby wrote: > This thread is about a new feature that I proposed which isn't yet committed > (logging leader_pid). But it raises a question which is immediately relevant > to pg_stat_activity.leader_pid, which is committed for v13. So feel free t

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-20 Thread Ajin Cherian
On Mon, Jul 20, 2020 at 11:16 PM Dilip Kumar wrote: > > > There was one warning in release mode in the last version in 0004 so > attaching a new version. > > -- > Regards, > Dilip Kumar > EnterpriseDB: http://www.enterprisedb.com Hello, I have tried to rework the patch which did the stats for

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Dilip Kumar
On Mon, Jul 20, 2020 at 10:14 PM Alvaro Herrera wrote: > > On 2020-Jul-20, Dilip Kumar wrote: > > > On Fri, Jul 17, 2020 at 4:16 PM Dilip Kumar wrote: > > > > So if the vacuum_tolerate_damage is set then in > > > all the cases in heap_prepare_freeze_tuple where the corrupted xid is > > > detected

Re: Added tab completion for the missing options in copy statement

2020-07-20 Thread Michael Paquier
On Sat, Jul 18, 2020 at 04:49:23PM +0530, vignesh C wrote: > Thanks Michael for the clarification, the patch looks fine to me. Thanks for the confirmation, committed. -- Michael signature.asc Description: PGP signature

Re: Parallel Seq Scan vs kernel read ahead

2020-07-20 Thread Amit Kapila
On Tue, Jul 21, 2020 at 8:06 AM k.jami...@fujitsu.com wrote: > > Thank you for the advice. I repeated the test as per your advice and average > of 3 runs > per worker/s planned. > It still shows the following similar performance results between Master and > Patch V2. > I wonder why there's no di

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Andrey M. Borodin
> 21 июля 2020 г., в 00:36, Alvaro Herrera > написал(а): > > >> FWIW we coped with this by actively monitoring this kind of corruption >> with this amcheck patch [0]. One can observe this lost page updates >> cheaply in indexes and act on first sight of corruption: identify >> source of the

Re: Wrong results from in_range() tests with infinite offset

2020-07-20 Thread Tom Lane
I wrote: > The other three cases where we'd hit NaNs are likewise symmetric with > non-NaN cases that'd return TRUE. Hence, I'm forced to the conclusion > that you've got it right above. I might write the code a little > differently, but const-TRUE-for-NaN-cases seems like the right behavior. > S

Re: NaN divided by zero should yield NaN

2020-07-20 Thread Tom Lane
Dean Rasheed writes: > On Thu, 16 Jul 2020 at 20:29, Tom Lane wrote: >> One thing that's not very clear to me is which of these spellings >> is preferable: >> if (unlikely(val2 == 0.0) && !isnan(val1)) >> if (unlikely(val2 == 0.0 && !isnan(val1))) > My guess is that the first would be

Re: expose parallel leader in CSV and log_line_prefix

2020-07-20 Thread Justin Pryzby
On Fri, Jul 17, 2020 at 05:32:36PM -0500, Justin Pryzby wrote: > On Fri, Jul 17, 2020 at 05:27:21PM -0400, Alvaro Herrera wrote: > > On 2020-Jul-17, Justin Pryzby wrote: > > > Ok, but should we then consider changing pg_stat_activity for consistency > > > ? > > > Probably in v13 to avoid changing

Re: pg_subscription.subslotname is wrongly marked NOT NULL

2020-07-20 Thread Tom Lane
Mopping this up ... the attached patch against v12 shows the portions of 72eab84a5 and 0fa0b487b that I'm thinking of putting into v10-v12. The doc changes, which just clarify that subslotname and srsublsn can be null, should be uncontroversial. The changes in pg_subscription.c prevent it from ac

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Andres Freund
Hi, On 2020-07-17 16:16:23 +0530, Dilip Kumar wrote: > The attached patch allows the vacuum to continue by emitting WARNING > for the corrupted tuple instead of immediately error out as discussed > at [1]. > > Basically, it provides a new GUC called vacuum_tolerate_damage, to > control whether to

Re: Local visibility with logical decoding

2020-07-20 Thread Andres Freund
Hi, On 2020-07-20 17:27:30 +0300, Marko Tiikkaja wrote: > It appears that when logical decoding sends out the data from the output > plugin, it is not guaranteed that the decoded transaction's effects are > visible on the source server. Is this the way it's supposed to work? At the moment the vi

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-07-20 Thread Robert Haas
On Tue, Jul 7, 2020 at 12:55 PM Andres Freund wrote: > What are you proposing? For now we could easily enough work around this > by just making it a on_proc_exit() callback, but that doesn't really > change the fundamental issue imo. I think it would be more correct for it to be an on_proc_exit()

Re: Stale external URL in doc?

2020-07-20 Thread Bruce Momjian
On Thu, Jul 9, 2020 at 09:51:51AM -0400, Tom Lane wrote: > Daniel Gustafsson writes: > > As a short term fix we should either a) remove these links completely or b) > > link to archived copies of the pages on archive.org; or c) find a more > > appropriate pages to link to. A quick search didn't

Re: [PATCH] Finally split StdRdOptions into HeapOptions and ToastOptions

2020-07-20 Thread Georgios Kokolatos
The following review has been posted through the commitfest application: make installcheck-world: not tested Implements feature: not tested Spec compliant: not tested Documentation:not tested Hi, thank you for the patch. It applies cleanly, compiles and passes check,

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Alvaro Herrera
On 2020-Jul-20, Andrey M. Borodin wrote: > I think the point here is to actually move relfrozenxid back. But the > mince can't be turned back. If CLOG is rotated - the table is > corrupted beyond easy repair. Oh, I see. Hmm. Well, if you discover relfrozenxid that's newer and the pg_clog files

Re: Binary support for pgoutput plugin

2020-07-20 Thread Tom Lane
Petr Jelinek writes: > On 20/07/2020 17:51, Tom Lane wrote: >> I'm fixing that, but even after that, there's a semantic problem: >> LOGICALREP_COLUMN_UNCHANGED is just a weak optimization, cf the code >> that sends it, in proto.c around line 480. colstatus will often *not* >> be that for columns

Re: Binary support for pgoutput plugin

2020-07-20 Thread Petr Jelinek
Hi, On 20/07/2020 17:51, Tom Lane wrote: Peter Geoghegan writes: Skink's latest run reports a failure that I surmise was caused by this patch: Yeah, I've just been digging through that. The patch didn't create the bug, but it allowed valgrind to detect it, because the column status array is

Re: Amcheck: do rightlink verification with lock coupling

2020-07-20 Thread Andrey M. Borodin
Hi! > 23 янв. 2020 г., в 00:59, Peter Geoghegan написал(а): > > On Fri, Jan 17, 2020 at 5:43 PM Peter Geoghegan wrote: >> I tried to come up with a specific example of how this could be >> unsafe, but my explanation was all over the place (this could have had >> something to do with it being Fr

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Andrey M. Borodin
> 20 июля 2020 г., в 21:44, Alvaro Herrera > написал(а): > >> I think we shall do that in some cases >> but IMHO it's not a very good idea in all the cases. Basically, if >> the xmin precedes the relfrozenxid then probably we should allow to >> update the relfrozenxid whereas if the xmin pre

Re: Default setting for enable_hashagg_disk

2020-07-20 Thread Tomas Vondra
On Mon, Jul 20, 2020 at 09:17:21AM -0400, Tom Lane wrote: Tomas Vondra writes: There's a minor problem here, though - these stats were collected before we fixed the tlist issue, so hashagg was spilling about 10x the amount of data compared to sort+groupagg. So maybe that's the first thing we sh

Re: Local visibility with logical decoding

2020-07-20 Thread Marko Tiikkaja
On Mon, Jul 20, 2020 at 7:36 PM Antonin Houska wrote: > Marko Tiikkaja wrote: > > > It appears that when logical decoding sends out the data from the output > > plugin, it is not guaranteed that the decoded transaction's effects are > > visible on the source server. Is this the way it's suppose

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Alvaro Herrera
On 2020-Jul-20, Dilip Kumar wrote: > On Fri, Jul 17, 2020 at 4:16 PM Dilip Kumar wrote: > > So if the vacuum_tolerate_damage is set then in > > all the cases in heap_prepare_freeze_tuple where the corrupted xid is > > detected, it will emit a warning and return that nothing is changed in > > the

Re: Local visibility with logical decoding

2020-07-20 Thread Antonin Houska
Marko Tiikkaja wrote: > It appears that when logical decoding sends out the data from the output > plugin, it is not guaranteed that the decoded transaction's effects are > visible on the source server. Is this the way it's supposed to work? Can you please share the test that indicates this beh

Re: Binary support for pgoutput plugin

2020-07-20 Thread Tom Lane
Peter Geoghegan writes: > Skink's latest run reports a failure that I surmise was caused by this patch: Yeah, I've just been digging through that. The patch didn't create the bug, but it allowed valgrind to detect it, because the column status array is now "just big enough" rather than being alw

Postgres-native method to identify if a tuple is frozen

2020-07-20 Thread Lawrence Jones
Hey all, *tl;dr: we're looking for an easy way to ask if a tuple is frozen from within a SQL query* We're trying to build a validation process around our CCD, in an attempt to validate that all data inside of Postgres has made it into our secondary store. Our plan is to build a small incremental

Re: Binary support for pgoutput plugin

2020-07-20 Thread Peter Geoghegan
On Sat, Jul 18, 2020 at 9:53 AM Tom Lane wrote: > I've pushed this patch, with a number of adjustments, some cosmetic > and some not so much (no pg_dump support!?). We're not quite > done though ... Skink's latest run reports a failure that I surmise was caused by this patch: ==722318== VALGRIN

Local visibility with logical decoding

2020-07-20 Thread Marko Tiikkaja
Hi, It appears that when logical decoding sends out the data from the output plugin, it is not guaranteed that the decoded transaction's effects are visible on the source server. Is this the way it's supposed to work? If so, would doing something like this in the output plugin be reasonable?

Re: Default setting for enable_hashagg_disk

2020-07-20 Thread Tom Lane
Tomas Vondra writes: > There's a minor problem here, though - these stats were collected before > we fixed the tlist issue, so hashagg was spilling about 10x the amount > of data compared to sort+groupagg. So maybe that's the first thing we > should do, before contemplating changes to the costing

Re: Mark btree_gist functions as PARALLEL SAFE

2020-07-20 Thread Alexander Korotkov
On Mon, Jul 20, 2020 at 3:18 PM Justin Pryzby wrote: > On Wed, Jul 15, 2020 at 03:26:24PM +0300, Alexander Korotkov wrote: > > On Thu, Jun 18, 2020 at 7:48 PM Winfield, Steven > > wrote: > > > Done - thanks again. > > > > This patch looks good to me. > > > > I'm going to push this if no objectio

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-20 Thread David Steele
On 7/20/20 6:02 AM, Fujii Masao wrote: On 2020/07/20 13:48, Fujii Masao wrote: On 2020/07/17 20:24, David Steele wrote: On 7/17/20 5:11 AM, Fujii Masao wrote: On 2020/07/14 20:30, David Steele wrote: On 7/14/20 12:00 AM, Fujii Masao wrote: The patch was no longer applied cleanly beca

Re: Mark btree_gist functions as PARALLEL SAFE

2020-07-20 Thread Justin Pryzby
On Wed, Jul 15, 2020 at 03:26:24PM +0300, Alexander Korotkov wrote: > On Thu, Jun 18, 2020 at 7:48 PM Winfield, Steven > wrote: > > Done - thanks again. > > This patch looks good to me. > > I'm going to push this if no objections. I marked as committed to make patch checker look healthier. Tha

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-20 Thread Ashutosh Bapat
On Fri, Jul 17, 2020 at 10:00 PM Etsuro Fujita wrote: > > On Sat, Jul 18, 2020 at 12:44 AM Ashutosh Bapat > wrote: > > On Fri, Jul 17, 2020 at 8:24 PM Etsuro Fujita > > wrote: > > > On Fri, Jul 17, 2020 at 1:56 AM Alexey Kondratov > > > wrote: > > > > However, there is an issue with aggregates

Re: problem with RETURNING and update row movement

2020-07-20 Thread Amit Kapila
On Thu, Jun 11, 2020 at 2:40 PM Amit Langote wrote: > > Hi, > > While working on [1], I came across a bug. > > Reproduction steps: > > create table foo (a int, b int) partition by list (a); > create table foo1 (c int, b int, a int); > alter table foo1 drop c; > alter table foo attach partition foo

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Dilip Kumar
On Sun, Jul 19, 2020 at 4:56 PM Andrey M. Borodin wrote: > > Hi Dilip! > > > > 17 июля 2020 г., в 15:46, Dilip Kumar написал(а): > > > > The attached patch allows the vacuum to continue by emitting WARNING > > for the corrupted tuple instead of immediately error out as discussed > > at [1]. > > >

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-20 Thread Fujii Masao
On 2020/07/20 13:48, Fujii Masao wrote: On 2020/07/17 20:24, David Steele wrote: On 7/17/20 5:11 AM, Fujii Masao wrote: On 2020/07/14 20:30, David Steele wrote: On 7/14/20 12:00 AM, Fujii Masao wrote: The patch was no longer applied cleanly because of recent commit. So I updated the p

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-20 Thread Dilip Kumar
On Fri, Jul 17, 2020 at 4:16 PM Dilip Kumar wrote: > > The attached patch allows the vacuum to continue by emitting WARNING > for the corrupted tuple instead of immediately error out as discussed > at [1]. > > Basically, it provides a new GUC called vacuum_tolerate_damage, to > control whether to

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-20 Thread Amit Kapila
On Mon, Jul 20, 2020 at 12:01 PM Dilip Kumar wrote: > > On Thu, Jul 16, 2020 at 4:25 PM Amit Kapila wrote: > > > > On Thu, Jul 16, 2020 at 12:23 PM Dilip Kumar wrote: > > > > > > On Wed, Jul 15, 2020 at 6:59 PM Amit Kapila > > > wrote: > > > > > > > > > > > > Let me know what you think of the

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-20 Thread Masahiko Sawada
On Sat, 18 Jul 2020 at 01:45, Fujii Masao wrote: > > > > On 2020/07/17 20:04, Masahiko Sawada wrote: > > On Fri, 17 Jul 2020 at 14:22, tsunakawa.ta...@fujitsu.com > > wrote: > >> > >> From: Masahiko Sawada > >> I have briefly checked the only oracle_fdw but in general I think that > >>> if an ex

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-20 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I also believe they do. But I'm concerned that some FDW needs to start > a transaction differently when using 2PC. For instance, IIUC MySQL > also supports 2PC but the transaction needs to be started with "XA > START id” when the transaction needs to be prepared. The transa